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ABSTRACT 


The  Maintenance  Climate  Assessment  Survey  (MCAS)  was  developed  to 
proactively  assess  factors  that  contribute  to  a  high  reliability  organization  and  strong 
safety  climate.  The  3rd  Marine  Air  Wing  (MAW),  which  was  seeking  to  proactively 
improve  its  safety  posture  requested  the  assistance  of  the  School  of  Aviation  Safety  at  the 
Naval  Postgraduate  School  to  examine  its  safety  climate.  Previous  studies  of  the  MCAS 
instrument  have  focused  on  the  items  and  their  relationship  to  the  HRO  based  model  of 
safety  effectiveness  components:  process  auditing,  reward  system,  quality  assurance,  risk 
management,  command  and  control,  and  communication/functional  relationships.  The 
present  effort  is  the  first  attempt  to  consider  the  relationship  between  item  component 
responses  and  demographic  item  responses.  It  evaluates  893  maintainer  responses  to  the 
MCAS  from  3rd  MAW  and  looks  for  measurable  effects  due  to  demographics.  This  study 
finds  that  the  regression  models  constructed  using  the  demographics  as  explanatory 
variables  have  very  little  utility  in  predicting  scores  for  the  components.  This  result 
allows  planners  the  relief  of  using  the  demographics  as  a  low  priority  issue. 
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EXECUTIVE  SUMMARY 


Naval  Aviation  is  a  hazardous  undertaking,  but  in  spite  of  its  inherent  risk,  its 
Class  A  Flight  Mishap  (FM)  rate  has  been  cut  in  half  for  each  decade  from  1950  to  1990. 
Over  the  last  decade,  however,  the  proportion  of  aircraft  losses  in  which  human  error  has 
been  cited  as  a  contributor  has  remained  relatively  constant.  To  address  human  factors 
issues  in  flight  mishaps,  the  Human  Factors  Quality  Management  Board  (HFQMB)  was 
established  in  1996.  By  using  Mishap  Data  Analysis  (MDA),  Organizational 
Benchmarking  (OB),  and  Command  Safety  Assessment  (CSA),  the  efforts  of  the 
HFQMB  resulted  in  a  significant  reduction  in  FM  incidence  from  the  perspective  of 
aircrew  operations. 

Although  human  error  in  maintenance  is  a  smaller  contributor,  it  has  been  shown 
to  be  a  factor  in  nearly  one  in  five  Class  A  FMs.  To  address  human  error  in  maintenance, 
the  Human  Factors  Analysis  and  Classification  System  -  Maintenance  Extension 
(HFACS-ME)  was  developed  to  classify  error  types  in  maintenance.  Since  Naval 
Aviation  is  shown  to  be  a  high  reliability  organization  (HRO)  defined  as  an  organization 
that  operates  in  hazardous  environment  with  less  than  its  fair  share  of  accidents,  it  shares 
common  characteristics  with  other  HROs.  These  common  characteristics  are  outlined  in 
the  Model  of  Organizational  Safety  Effectiveness  (MOSE)  and  are  process  auditing, 
quality,  reward  system,  risk  management  and  command  and  control.  Military  aviation  has 
communication/functional  relationships  as  a  sixth  component.  The  augmented  MOSE  is 
the  basis  of  the  Maintenance  Climate  Assessment  Survey  (MCAS),  which  is  used  to 
evaluate  the  organizational  safety  climate  from  the  perspective  of  the  maintenance 
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personnel.  The  MCAS  consists  of  six  demographic  items  and  43  perception  items.  Each 
of  the  43  perception  items  maps  into  a  single  component  of  the  augmented  MOSE.  These 
questions  are  collapsed  into  six  component  scores  for  each  respondent. 

Although  MCAS  has  been  shown  to  be  an  effective  tool  for  evaluating  the  safety 
environment  in  a  maintenance  organization,  demographic  factors  and  their  potential 
relationship  with  maintainer  responses  have  not  been  investigated.  This  study  evaluates 
MCAS  responses  from  894  maintenance  personnel  of  the  3rd  MAW,  and  looks  at  how  the 
demographic  factors  of  maintenance  personnel  might  be  biasing  the  component  scores  of 
the  MCAS.  The  results  of  this  thesis  are  intended  to  further  refine  MCAS  demographic 
factors  and  provide  Squadron  Commanders  with  insight  into  the  construct  of  their 
maintenance  personnel. 

The  component  scores  are  fitted  using  the  demographics  as  explanatory  factors. 
Univariate  analysis  is  performed  for  each  component  using  simple  models  without 
interaction  and  also  with  models  using  two-factor  interactions.  These  models  are  then 
simplified  in  order  to  reduce  the  number  of  terms  to  a  more  manageable  level. 

The  results  of  this  thesis  show  that  up  to  two-factor  interaction,  the  demographic 
factors  of  the  MCAS  poorly  account  for  the  variance  in  the  responses.  The  reliance  on 
subjective  perception  in  the  scoring  is  the  cause  of  the  large  amount  of  variance.  Since 
variance  cannot  be  explained  by  the  demographic  factors,  the  MCAS  appears  to  be 
demographically  unbiased.  Input  from  subject  matter  experts  is  used  to  refine  the 
demographic  factors.  These  revised  factors  are  more  usable  for  Squadron  Commanders  in 
that  they  provide  more  insight  into  the  make  up  of  the  maintenance  organization. 
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I.  INTRODUCTION 


A.  OVERVIEW 

Naval  Aviation  is  a  hazardous  undertaking,  but  in  spite  of  its  inherent  risk,  its 
Class  A  Flight  Mishap  (FM)  rate  has  been  cut  in  half  for  each  decade  from  1950  and 
1990  (Naval  Safety  Center,  1997).  Class  A  Mishaps  are  defined  as  Naval  aircraft 
incidents  resulting  in  death,  permanent  disability,  or  property  loss  or  damage  in  excess  of 
one  million  dollars  (OPNAV  3750.6Q,  1989).  A  flight  mishap  (FM)  is  defined  as  those 
mishaps  in  which  there  is  $10,000  or  greater  DoD  aircraft  damage  or  loss  of  a  DoD 
aircraft,  and  intent  for  flight  for  DoD  aircraft  existed  at  the  time  of  the  mishap.  Other 
property  damage,  injury,  or  death  may  or  may  not  have  occurred.  Naval  Aviation 
consistently  maintains  high  levels  of  operability  coupled  with  less  than  its  fair  share  of 
accidents  (Goodrum,  1999).  Naval  Aviation  also  possesses  the  requisite  characteristics  of 
a  high  reliability  organization  (HRO):  process  auditing,  reward  system,  quality,  risk 
management,  and  command  and  control.  For  these  reasons,  Roberts  (1988)  labeled  Naval 
Aviation  an  HRO. 

Although  Naval  Aviation  is  successful  in  reducing  its  Class  A  FM  rate,  over  the 
last  decade  the  proportion  of  aircraft  losses  in  which  human  error  has  been  cited  as  a 
contributor  has  remained  relatively  constant  at  four  of  five  FMs  (Naval  Safety  Center, 
2000).  In  1996,  a  Human  Factors  Quality  Management  Board  (HFQMB)  is  established 
after  17  Class  A  FMs  occurred  in  only  75  days,  climaxing  when  a  Navy  F-14  crashes  into 
a  Nashville,  TN  neighborhood,  to  address  human  factors  issues  related  to  mishaps 
(Nutwell  &  Sherman,  1997).  The  goal  of  the  HFQMB  is  to  cut  the  current  Class  A  FM 
rate  due  to  human  error  in  half  by  year  2000  (HFQMB  Charter,  1996).  The  HFQMB 
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adopts  three  approaches  to  identify  and  target  factors  contributing  to  human  error:  1) 
Mishap  Data  Analysis  (MDA),  2)  Organizational  Benchmarking  (OB),  and  3)  Command 
Safety  Assessment  (CSA). 

MDA  establishes  the  development  of  HFACS,  which  is  used  to  identify  and 
prioritize  human  factors  contributors  to  FMs.  Among  others,  it  determines  inadequate 
supervision  and  aircrew  violations  are  significant  contributors  (Shappel  &  Wiegman, 
1997).  Using  OB  which  explores  programs  which  influence  aircrew  performance,  the 
HFQMB  determines  use  of  feedback  mechanisms  in  commercial  airlines  improve  crew 
resource  management  training  benefits  (Nutwell  &  Sherman,  1997).  Finally,  a  CSA 
survey,  based  on  a  model  of  HROs,  is  developed  to  determine  a  command’s  safety 
posture  from  an  aircrew  perspective.  This  survey  finds  that  55%  of  the  Navy  respondents 
and  65%  of  the  Marine  Corps  respondents  feel  that  their  commands  are  committed 
beyond  what  available  resources  can  provide  (Ciavarelli  &  Figlock,  1997).  These 
combined  efforts  make  significant  progress  toward  the  HFQMB ’s  goal  as  evidenced  by 
fiscal  year  1999  being  the  safest  year  in  Naval  Aviation  history  in  terms  of  Class  A  FM 
rate. 

Maintenance  is  shown  to  be  a  contributing  factor  in  nearly  one  in  five  Class  A 
FMs  (Naval  Safety  Center,  2000).  Additionally,  during  FY90-97,  Class  C  FMs  account 
for  75%  of  all  maintenance  related  mishaps  (MRMs).  Maintenance  is  one  area  where 
hazards  can  be  controlled  and  risk  can  be  managed  while  an  aircraft  is  on  the  ground. 
Much  work  is  done  in  the  field  of  human  factors  in  maintenance  safety  for  commercial 
airlines  (“Human  Factors  in,”  2000).  In  1988,  the  Aviation  Safety  Research  Act  (ASRA) 
mandates  close  study  of  aging  aircraft  structures  and  human  factors  affecting  safety 
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(“History,”  2000).  In  the  spirit  of  this  mandated  study,  Boeing  finds  that  incomplete 
installation  (34%),  damaged  on  installation  (15%),  improper  installation  (11%)  and 
equipment  not  installed/missing  (11%)  were  the  top  contributors  in  maintenance  error 
(Komamiski,  2000).  This  investigation  and  classification  of  types  of  human  error  in 
maintenance  leads  to  the  development  of  Boeing’s  Maintenance  Error  Decision  Aid 
(MED A),  a  system  that  aids  operators  and  maintainers  in  the  investigation  and  mitigation 
of  maintenance  related  errors  (Allen,  Rankin,  &  Sargent,  1998). 

The  ASRA  is  one  of  the  precursors  for  the  FAA’s  current  goal  to  reduce  the  fatal 
accident  rate  80%  by  2007  as  compared  to  1994-1996  baseline  data  (FAA,  1998).  Several 
key  initiatives  are  the  stepping  stones  for  this  ultimate  goal:  1)  the  development  of  a 
maintenance  resource  management  system;  2)  establishment  of  new  training 
requirements;  3)  implementation  of  technical  advances  in  aircraft  maintenance  at  repair 
stations;  4)  enforcement  of  safety  recommendations  from  the  National  Transportation 
Safety  Board  (NTSB);  and  5)  recommendations  for  aging  systems  maintenance.  Little 
work  until  recently  is  done  for  military  aviation,  and  recent  efforts  involve  studying 
forms  of  maintenance  error  (Schmorrow,  1998)  and  the  perceived  maintenance  safety 
climate  (Baker,  1998).  From  a  proactive  perspective,  efforts  must  be  made  to  continue 
developing  assessment  tools  to  identify  potential  areas  for  risk  management  and  control 
of  conditions  before  a  mishap  occurs. 

Using  the  Maintenance  Climate  Assessment  Survey  (MCAS),  Goodrum  (1999) 
and  Oneto  (1999)  are  able  to  show  the  prototype  survey  effectively  evaluates  a 
maintainer’s  perception  of  safety  in  maintenance  operations.  They  also  further  refine  the 
MCAS  into  a  present  43  question  format.  One  aspect  of  the  MCAS  yet  to  be  addressed  is 
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the  demographic  categories  to  determine  their  potential  relationship  with  maintainer 
responses.  Given  the  structure  of  maintenance  organizations  within  aircraft  communities 
vary,  it  is  unlikely  that  any  pair  of  samples  will  have  equal  proportions  of  all 
demographic  variables.  It  is  anticipated  that  individual  demographic  characteristics  may 
influence  MCAS  responses  and  are  therefore  potentially  biasing  the  results.  By 
understanding  the  effects  of  the  demographics,  one  can  understand  if  they  impact  their 
organization’s  safety  climate. 

B.  BACKGROUND 

The  3rd  Marine  Air  Wing  (MAW)  is  a  combat-ready  expeditionary  aviation  force 
capable  of  short-notice  worldwide  employment  to  Marine  Air  Ground  Task  Force 
(MAGTF)  fleet  and  unified  commanders.  It  is  composed  of  28  squadrons  divided  into 
four  Marine  Air  Groups  (MAGs)  based  in  Southern  California  and  Arizona.  Each  MAG 
has  its  own  combat  mission:  MAG  11  provides  air  support  to  MAGTF  commanders; 
MAG- 13  provides  close-air  support,  conducts  armed  reconnaissance,  and  assumes 
limited  air-defense  roles;  MAG- 16  transports  and  resupplies  Marine  air  and  ground  units; 
and  MAG-39  provides  utility  helicopter  support,  close-in  fire  support,  fire  support 
coordination,  aerial  reconnaissance,  observation  and  forward  air  control  in  aerial  and 
ground  escort  operations  during  ship-to-shore  movement  and  subsequent  operations 
ashore.  The  aircraft  used  in  these  missions  are  AH-lWs,  UH-lNs,  CH-53s,  CH-46Es, 
F/A-18DS,  F/A-18s,  AV-8s  and  C-130Ts. 

From  1990  to  1996,  maintenance  is  a  causal  factor  in  17  percent  of  all  Naval 
Aviation  class  A  FMs  (Naval  Safety  Center,  1997).  From  April  1997  to  July  1999, 
maintenance,  maintenance  personnel  or  maintenance  depot  is  cited  as  a  causal  factor  in 
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14  FMs  (eight  class  C  FMs,  four  class  B  FMs,  and  two  class  A  FMs)  experienced  by  3rd 
MAW.  The  Commander  of  3rd  MAW  requests  the  assistance  of  the  School  of  Aviation 
Safety  (SAS)  at  the  Naval  Postgraduate  School,  which  in  turn  provides  safety  and  risk 
management  training  to  personnel,  mishap  data  analysis,  and  administration  of  safety 
climate  surveys  to  help  locate  problems  in  the  organization. 

C.  PROBLEM  STATEMENT 

Human  error  in  aviation  is  an  issue  that  needs  to  be  addressed,  and  it  is  recognized 
that  the  organization  has  an  impact  on  factors  that  lead  to  it.  Organizations  that  possess 
the  attributes  of  a  HRO  tend  to  generate  environments  conducive  to  the  reduction  or 
control  of  human  error  and  consequently  experience  fewer  mishaps.  Organizations 
aspiring  towards  the  reduction  of  mishaps  need  to  assess  their  safety  posture  as  it  relates 
to  the  attributes  of  HROs.  The  School  of  Aviation  Safety  at  the  Naval  Postgraduate 
School  has  developed  surveys  to  assess  HRO  characteristics  in  the  operational 
environment  for  aircrew  and  maintenance  personnel. 

The  3rd  MAW  in  an  attempt  to  improve  its  safety  posture  enlists  to  have  the 
School  of  Aviation  Safety  employ  the  MCAS  survey  to  assess  maintainer  perception  of 
HRO  characteristics  in  its  recent  operations.  These  results  are  revealing,  however  in  order 
to  provide  for  better  interpretation  of  the  results,  an  exploration  of  the  demographic 
variables  is  in  order.  This  will  help  commanders  to  target  more  effectively  specific  areas 
of  the  organization  that  require  attention. 

The  current  version  of  the  MCAS  is  administered  to  the  3rd  MAW  during  the  last 
half  of  1999.  Using  statistical  methods,  the  collected  data  is  analyzed  to  assess 
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differences  in  responses  that  are  correlated  to  differences  in  demographics.  This  thesis 
explores  the  following  questions: 

1.  Are  there  measurable  demographic  effects  to  the  responses  on  the  MCAS? 

2.  Is  there  enough  information  in  the  demographics  to  be  used  in  an  adjustment 
process  of  the  overall  scores? 

3.  Can  the  MCAS  be  refined  further  to  either  collapse  or  expand  demographic 
factors? 

D.  SCOPE  AND  LIMITATIONS 

Active  duty  U.S.  Marine  Corps  Squadrons  maintenance  personnel  of  the  3rd 
MAW  are  surveyed  during  the  fall  of  1999.  Only  those  squadrons  with  a  representative 
number  of  respondents  are  used  in  the  survey.  Chapter  II  provides  a  basis  for 
understanding  human  error,  organizational  safety  culture,  high  reliability  organizations 
and  the  assessment  of  a  safety  climate.  Chapter  III  presents  a  discussion  of  the 
methodology  used  in  this  study.  Results  of  data  analysis  are  presented  in  Chapter  IV. 
Chapter  V  summarizes  previous  chapters  and  provides  conclusions  and  recommendations 
as  they  relate  to  the  material. 

E.  DEFINITIONS 

This  thesis  uses  the  following  definitions  (DON,  1989): 

Naval  Aircraft.  Refers  to  U.S.  Navy,  Naval  Reserve,  U.S.  Marine  Corps,  and  U.S. 
Marine  Corps  Reserve  aircraft. 

Mishap.  A  Naval  Aviation  mishap  is  an  unforeseen  or  unplanned  event  that  directly 
involves  naval  aircraft,  which  result  in  $10,000  or  greater  cumulative  damage  to  naval 
aircraft  or  personnel.  The  mishap  is  further  divided  into  three  classes  based  on  the 
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amount  of  damage  to  the  aircraft,  property  and  personnel  injury.  The  following  are 
the  definitions  of  the  three  classes: 

a.  Class  A.  A  mishap  in  which  the  total  cost  of  property  damage  (including  all 
aircraft  damage)  is  $1,000,000  or  greater;  or  a  naval  aircraft  is  destroyed  or 
missing;  or  any  fatality  or  permanent  total  disability  of  a  person  occurs  with  direct 
involvement  of  naval  aircraft. 

b.  Class  B.  A  mishap  in  which  the  total  cost  of  property  damage  (including  all 
aircraft  damage)  is  $200,000  or  more  but  less  than  $1,000,000  and/or  a  permanent 
partial  disability,  and  or  the  hospitalization  of  five  or  more  personnel. 

c.  Class  C.  A  mishap  in  which  the  total  cost  of  property  damage  (including  all 
aircraft  damage)  is  $10,000  or  more  but  less  than  $200,000  and/or  injury  results  in 
one  or  more  lost  workdays. 

Mishap  rate.  The  total  number  of  Class  A,B  and  C  mishaps  per  100,000  flight  hours. 
MCAS.  A  43-question  survey  used  to  gain  insight  into  the  maintenance  community’s 
perception  concerning  aviation  mishaps  within  the  Navy  and  Marine  Corps. 
HFACS-ME.  A  taxonomic  system  used  to  classify  causal  factors  that  contribute  to 
maintenance  related  mishaps. 

HRO.  High-Reliability  Organization,  is  an  organization  that  operates  in  a  hazardous 
environment,  yet  produces  very  low  rate  of  accidents  and  incidents,  operating  effectively 
and  safely  and  having  the  characteristics  of  leadership,  sound  management  policies, 
procedure  standardization,  adequacy  of  resources  and  staffing,  a  defined  system  for  risk 
management,  and  other  factors. 
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II.  LITERATURE  REVIEW 


A.  HUMAN  ERROR 

Reason  (1990)  defines  error  as  a  planned  sequence  that  fails  to  achieve  its 
intended  outcome  in  the  absence  of  external  influence.  He  (1997)  later  describes  error 
types  as  active  or  latent.  Where  the  effects  of  active  errors  are  often  immediate  and 
confined,  latent  conditions  lie  dormant  until  set  off  by  a  chain  of  local  events  and  can  be 
contributing  factors  in  a  variety  of  failures.  This  model  of  latent  conditions  and  active 
failures  is  adopted  by  the  Naval  Safety  Center  to  investigate  Class  A  FMs  with  respect  to 
aircrew  error,  and  was  the  basis  for  the  development  of  Human  Factors  Analysis  and 
Classification  System  (HFACS)  (Shappel  &  Wiegman,  1997). 


I  Activity 

‘Hands  On’ 

■KifllifgW 

Normal  control 

Low 

Moderate 

HHE39H 

Emergency  control 

Moderate 

Low 

|  Maintenance-related 

■■ 

HES3HI 

MEM3HI 

Table  1.  Likelihood  of  Performance  Problems  in  Given  Activities. 


Reason  (1997)  also  models  human  error  in  the  scope  of  universal  human  activities 
(see  Table  1)  and  the  likelihood  of  performance  problems  within  each  of  these  types  of 
activities.  With  HFACS,  the  Naval  Safety  Center  is  able  to  address  human  error  in 
normal  control  and  emergency  control  conditions.  But  Reason  asserts  that  maintenance  is 
the  area  with  the  highest  likelihood  of  human  error  because  maintenance  related  activities 
are  almost  exclusively  reliant  on  human  performance  in  the  three  areas  of  hands  on, 
criticality  and  frequency.  Even  with  progress  in  technology,  human  fallibility  remains 
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constant  (Reason,  1997)  and  with  the  frequency  of  planned  maintenance  compounded 
with  the  many  pairs  of  fallible  human  hands  working  on  exceptionally  complicated 
systems,  Reason’s  model  is  a  chilling  prediction  of  3rd  MAW’s  situation. 

To  address  the  maintenance  related  activity  as  a  contributor  to  the  total  of  human 
errors  in  FMs,  Schmidt,  Schmorrow  and  Hardee  (1998)  extend  HFACS  to  specifically 
address  the  maintenance  component  of  accident  causation  with  the  HFACS -Maintenance 
Extension  (ME).  The  HFACS-ME  expands  upon  Reason’s  model  of  latent  states  and 
psychological  precursors  to  unsafe  acts.  Reason  (1990;  1997)  differentiates  these  latent 
states  in  that  they  create  the  potential  for  human  error.  HFACS-ME  classifies  latent  states 
in  the  maintenance  environment,  with  three  levels  of  error  causation  under  four 
categories  of  conditions.  The  first  order  conditions  are  broad  precursor  categories 
(supervisory  conditions,  maintainer  conditions,  working  conditions  and  maintainer  acts) 
that  are  further  divided  into  more  specific  precursors  to  human  error  of  the  second  and 
third  order  (see  Table  2). 

The  causes  of  human  error  are  many.  Wickens,  Gordon  and  Lui  (1997)  state  that 
human  error  can  be  induced  by  “inattentiveness,  poor  work  habits,  lack  of  training,  poor 
decision  making,  personality  traits,  social  pressures,  and  so  forth”  (p.  427-428).  Redmill 
&  Rajan  (1997)  note  that  a  common  component  in  accidents  is  a  worker’s  loss  of 
concentration  which  can  be  caused  by  “boredom,  disinterest,  distraction,  or  attempt  to  do 
two  or  more  things  at  once”  (p.12).  Wickens,  Gordon  and  Lui  (1997)  assert  that  the  most 
common  type  of  maintenance  related  error  is  that  of  omission.  Considering  that  the  nature 
of  planned  maintenance  is  to  frequently  disassemble,  inspect  then  reassemble 
components,  it  is  clear  to  see  the  high  probability  of  human  error  in  maintenance  (Reason 
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1997).  But  as  Perrow  and  Reason  point  out,  the  errors  themselves  are  seldom  isolated,  but 
often  a  single  event  in  a  causal  chain  (Perrow,  1984),  or  encouraged,  or  at  least  not 
prohibited  from  occurring  by  latent  conditions  in  the  organization  (Reason,  1997). 


First  Order 

Second  Order 

Third  Order 

Supervisory  Conditions 

Unforeseen 

Hazardous  Operations 
Inadequate  Documentation 
Inadequate  Design 

Squadron 

Inadequate  Supervision 
Inappropriate  Operations 

Failed  to  Correct  Problem 
Supervisory  Violation 

Maintainer  Conditions 

Medical 

Mental  State 

Physical  State 

Physical/Mental  Limitation 

Crew  Coordination 

Communication 

Assertiveness 

Adaptability/Flexibility 

Readiness 

Preparation/Training 

Qualification/Certification 

Violation 

Working  Conditions 

Environment 

Lighting/Light 
Exposure/Weather 
Environmental  Hazards 

Equipment 

Damaged 

Unavailable 

Dated/Uncertified 

Workspace 

Confining 

Obstructed 

Inaccessible 

Maintainer  Acts 

Error 

Attention 

Memory 

Rule/Knowledge 

Skill 

Violation 

Routine 

Infraction 

Exceptional 

Table  2.  HFACS-ME  Levels  of  Error  Causation. 
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B.  ORGANIZATIONAL  SAFETY  CULTURE 


1.  Definition 

Organizational  culture  is  defined  as  shared  values  and  beliefs  that  interact  with  an 
organization’s  structures  and  control  systems  to  produce  behavioral  norms  (Uttal,  1983). 
All  organizations  have  their  own  engineered  culture  whether  good  or  bad.  A  safety 
culture  is  ideal  for  complex  organizations  and  is  defined  as  the  product  of  individual  and 
group  values,  attitude,  competencies,  and  patterns  of  behavior  that  determine  the 
commitment  to,  and  the  style  and  proficiency  of,  an  organization’s  health  and  safety 
programmes  (Booth,  1993). 

2.  Composition 

Redmill  and  Raj  an  (1997)  state  that  there  are  three  general  aspects  of  safety 
culture:  awareness,  commitment  and  competence.  Awareness  must  be  present  in  all 
aspects  of  design,  management  and  decision  making.  When  a  safety  mishap  does  occur, 
it  is  commitment  that  drives  the  leadership  of  the  organization  to  investigate  and  locate 
contributing  factors  and  take  immediate  action  to  prevent  another  occurrence. 
Competence  is  a  combination  of  education,  training,  professionalism  and  personality 
traits  that  are  appropriate  for  a  given  task  or  job  (Redmill  &  Rajan,  1997). 

Reason  (1997)  prefers  the  term  informed  culture,  and  he  divides  informed  culture 
into  four  subcultures:  reporting  culture,  just  culture,  flexible  culture  and  learning  culture. 
Reporting  culture  is  “an  organizational  climate  in  which  people  are  prepared  to  report 
their  errors  and  near-misses.”  Just  culture  is  an  “atmosphere  of  trust  in  which  people  are 
encouraged,  even  awarded,  for  provided  essential  safety-related  information  -  but  in 
which  they  are  also  clear  about  where  the  line  must  be  drawn  between  acceptable  and 
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unacceptable  behavior.”  Flexible  culture  involves  “shifting  from  the  conventional 
hierarchical  mode  to  a  flatter  professional  structure,  where  control  passes  to  task  experts 
on  the  spot,  and  then  reverts  back  to  the  traditional  bureaucratic  mode  once  the 
emergency  has  passed.  Such  adaptability  is  an  essential  feature  of  the  crisis-prepared 
organization.”  Learning  culture  is  “the  willingness  and  the  competence  to  draw  the  right 
conclusions  from  its  safety  information  system,  and  the  will  to  implement  major  reforms 
when  their  need  is  indicated.” 

Safety  culture  has  powerful  effects.  First,  it  is  self  perpetuating  where  workers 
learn  from  each  other  and  encourage  each  other  to  work  in  a  manner  consistent  with  the 
organization’s  safety  culture.  People  are  quick  to  follow  the  example  of  coworkers,  even 
if  this  means  a  lack  of  vigilance  in  safety  (Redmill  &  Rajan,  1997).  Wogalter,  Allison,  & 
McKenna  (1989)  assert  that  “people  are  extremely  susceptible  to  social  norms;  they  are 
likely  to  engage  in  safe  or  unsafe  behaviors  to  the  extent  that  others  around  them  do  so.” 
Safety  culture  is  also  self  preserving  as  new  workers  learn  to  identify  acceptable  methods 
of  accomplishing  work  and  are  able  to  pass  those  standards  along  to  new  employees, 
good  or  bad  (Redmill  &  Rajan,  1997). 

C.  HIGH  RELIABILITY  ORGANIZATIONS 

1.  Definition 

Roberts  (1990)  and  Libuser  (1994)  explain  that  High-Reliability  Organizations 
(HROs),  organizations  that  operate  in  a  hazardous  environment,  yet  produce  very  low 
rates  of  accidents  and  incidents,  which  operate  effectively  and  safely  have  certain  key 
characteristics  in  common.  Reason  (1997)  calls  them  organizations  with  less  than  their 
fair  share  of  accidents,  and  “highly  complex,  technology-intensive  organizations  that 
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must  operate,  as  far  as  humanly  possible,  to  a  failure-free  standard.”  He  also  explains 
that  HROs  manage  complex  technologies  that  have  very  low  tolerance  for  error,  yet 
maintain  the  flexibility  to  successfully  function  in  environments  of  extreme  intensity. 

2.  Characteristics  of  HROs 

Examples  of  HROs  are  the  nuclear  power  industry,  petrochemical  industry,  and 
airline  industry.  Additionally,  Figlock  (1998)  identifies  Naval  aviation  is  an  HRO. 
Although  diverse  in  purpose,  Roberts  and  Libuser  believe  these  organizations  share 
several  common  characteristics:  leadership  style,  management  policies,  procedures 
standardization,  superior  training,  a  reward  system  that  recognizes  safety  achievement, 
adequacy  of  resources  and  staffing,  effective  management  of  risks  associated  with 
hazardous  operations,  and  other  factors. 

HROs  have  a  requisite  variety.  As  Weick  (1987)  states,  having  diverse  people 
from  diverse  backgrounds  and  experiences  builds  requisite  variety  that  is  required  for 
relatively  simple  humans  to  operate  complex  systems.  Additionally,  this  diversity  is 
essential  in  problem  solving,  as  individuals  will  approach  the  same  problem  uniquely,  so 
that  the  collective  contribution  is  greater  than  any  one  individual’s  input. 

HROs  typically  exhibit  a  high  degree  of  training.  Weick  (1987)  notes  that 
“training  for  the  operation  of  high  reliability  systems  is  often  tough  and  demanding  so 
that  the  faint  of  heart  and  the  incompetent  are  weeded  out.”  This  is  because  HROs  are  not 
afforded  the  luxury  of  trial  and  error.  Training  is  often  in  the  form  of  simulation  and 
stories.  Stories  have  a  big  affect  on  the  reliability  of  an  HRO  by  lending  experience  to  the 
inexperienced: 
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The  basic  idea  is  that  a  system  which  values  stories,  story  tellers,  and 
storytelling  will  be  more  reliable  than  a  system  that  derogates  these 
substitutes  for  trial  and  error.  A  system  that  values  stories  and  storytelling 
is  potentially  more  reliable  because  people  know  more  about  their  system, 
know  more  of  the  potential  errors  that  might  occur,  and  they  are  more 
confident  that  they  can  handle  those  errors  that  do  occur  because  they 
know  that  other  people  have  already  handled  similar  errors  (Weick,  1987, 

P-  1 13). 

By  sharing  the  experiences  of  skilled  personnel,  novices  do  not  have  to  learn  from  their 

own  mistakes  and  are  also  granted  the  insight  of  the  skilled. 

HROs  appear  bureaucratic  and  uneventful  on  the  surface.  There  is  a  strict  chain  of 

command  in  place  that  dictates  policy,  procedure  and  environment.  This  strong 

centralization  is  apparent  during  periods  of  relatively  low  intensity,  but  as  intensity  of 

operating  increases,  the  true  nature  of  the  HRO  is  revealed,  where  flexibility,  delegation, 

improvisation  and  technical  expertise  dominate  (Reason,  1997;  Weick,  1987).  This  is 

how  HROs  can  be  simultaneously  centralized  and  decentralized.  Responsibility  and 

judgment  remain  centralized  while  creativity,  improvisation  and  unsupervised  problem 

solving  become  decentralized  in  environments  of  high  intensity  (Weick,  1987). 

Reliability  in  HROs  is  another  deceptive  aspect.  Weick  (1987)  calls  reliability  a 

“dynamic  non-event,”  meaning  a  great  deal  of  effort  goes  into  ensuring  nothing  happens. 

This  dynamicism  is  based  on  the  belief  that  reliability  is  fleeting  and  systems  tend  to 

move  to  states  of  unreliability  unless  constantly  maintained: 

Part  of  the  mindset  for  reliability  requires  chronic  suspicion  that  small 
deviations  may  enlarge,  a  sensitivity  that  may  be  encouraged  by  a  more 
dynamic  view  of  reliability  (Weick,  1987,  p.  1 19). 

It  is  the  organizational  culture  for  personnel  to  look  for  problems  before  they  happen. 

Weick  continues  that  because  of  the  invisibility  of  the  dynamics  behind  reliability,  there 
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is  a  perception  that  reliability  is  easily  achievable  and  is  only  noticed  in  the  presence  of  a 
breakdown. 

D.  ASSESSING  SAFETY  CLIMATE 

1.  Safety  Space 

Reason  (1997)  states  that  organizations  can  be  mapped  into  a  safety  space  which 
is  a  continuum  of  degrees  of  susceptibility  to  accidents  (see  Figure  1).  Organizations  with 
higher  resistance  will  generally  have  fewer  mishaps  while  organizations  with  higher 
vulnerability  will  generally  experience  more  mishaps.  No  organization  is  immune. 
Chance,  unforeseen  circumstances,  failures  in  defenses  and  human  error  can  cause  even 
the  most  resistant  organizations  to  experience  accidents.  Within  the  safety  space,  currents 
tend  to  push  organizations  away  from  the  extremes  of  resistance  or  vulnerability  and 
toward  the  center,  a  compromise  between  the  two.  If  an  organization  has  the  desire  to 
become  more  resistant,  it  must  swim  upstream. 


Figure  1.  Organizational  Safety  Space. 
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The  effort  required  to  move  in  the  direction  of  increased  resistance  must  be  put 
into  reactive  and  proactive  measures.  Reason  (1997)  contends  that  investigating  mishaps 
to  find  causal  factors  to  be  addressed  is  not  even  half  the  battle.  To  effectively  move  the 
organization,  mishap  investigation  must  be  coupled  with  the  identification  of  conditions 
needing  correction,  and  regular  checks. 

2.  MOSE  and  MCAS 

Libuser’s  (1994)  current  Model  of  Organizational  Safety  Effectiveness  (MOSE)  is 
based  on  work  by  Roberts  and  is  a  categorization  of  the  common  characteristics  of 
HROs.  These  characteristics  are  mapped  into  five  components:  1)  Process  Auditing  (PA) 
-  checks  by  members  to  identify  hazards;  2)  Reward  System  (RS)-  expected  rewards  or 
disciplinary  action  used  to  shape  behavior;  3)  Quality  Assurance  (QA)-  promotion  of 
quality  performance;  4)  Risk  Management  (RM)-  system  to  identify  hazards  and  control 
operational  risks;  and  5)  Command  and  Control  (CC)-  safety  climate,  leadership 
effectiveness  /policies,  and  procedures  for  mitigating  risks.  These  components  are  very 
similar  to  the  aspects  of  Reason’s  (1997)  informed  culture  (Table  3). 


Libuser’s  MOSE  Components 

Reason ’s  Informed  Culture 

Process  Auditing  (PA). 

Leaning  Culture 

Reward  System  (RS) 

Just  Culture 

Quality  Control  (QA) 

Reporting  Culture 

Risk  Management  (RM) 

Flexible  Culture 

Command  and  Control  (CC) 

Flexible  Culture 

Table  3.  Comparison  of  Libuser’s  MOSE  and  Reason’s  Informed  Culture. 
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Ciavarelli  and  Figlock  (1997)  adapt  the  MOSE  for  use  in  Naval  Aviation  using 
practices  and  terminology  of  that  environment  and  develop  the  Command  Safety 
Assessment,  a  survey  that  addresses  each  of  the  MOSE  categories  from  the  viewpoint  of 
the  aircrewman.  This  survey  is  administered  to  1,254  aviators  revealing  that 
organizational  and  supervisory  issues  are  seen  by  aircrewmen  as  impacting  flight  safety. 

The  Maintenance  Climate  Assessment  Survey  (MCAS)  is  the  product  of  the 
implementation  of  the  MOSE  and  CSA  in  a  maintenance  context.  Baker  (1998)  starts  by 
reducing  155  candidate  questions  to  67  items  that  specifically  addressed  aviation 
maintenance.  Augmenting  Libuser’s  (1994)  five  category  MOSE  model  with  a  sixth 
category,  Communication/Functional  Relationships,  Baker  (1998)  modifies  the  CSA  to 
look  at  aviation  safety  from  the  point  of  view  of  the  maintenance  person.  Using 
regression  techniques,  Baker  is  able  to  further  reduce  the  survey  into  a  compact  35  item 
form,  with  almost  all  questions  mapped  to  a  single  category  of  the  augmented  MOSE. 

Goodrum  (1999)  and  Oneto  (1999)  show  that  the  MCAS  is  a  valid  tool  to 
accurately  assess  an  aviation  maintenance  environment,  but  note  that  some  items  in  the 
survey  need  restructuring.  Oneto  notes  that  these  items  address  more  than  one  category  of 
the  MOSE.  Their  inputs  help  change  the  MCAS  to  its  current  43  item  format  (see 
Appendix  A).  While  Goodrum  and  Oneto  are  able  to  show  content  validity  in  the  MCAS, 
there  is  much  left  to  examine.  Since  there  is  no  known  or  accepted  measure  for  MCAS 
results  and  providing  feedback  to  the  concerned  squadrons,  an  effort  is  underway  to 
explore  concurrent  validity  in  the  survey  on  a  per  question  basis  (Schmidt,  personal 
communication).  Questions  that  have  a  low  response  mean  are  noted  as  areas  that  need 
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attention,  corresponding  to  a  particular  category  of  the  MOSE,  and  which  part  of  the 
HRO  needs  closer  examination. 

Additionally,  by  looking  at  squadron  mean  scores  to  the  survey  and  available 
mishap  data,  Harris  (personal  communication)  is  looking  at  the  MCAS  predictive  validity 
in  the  incidence  of  mishaps  within  a  squadron  based  on  adjusted  mean  scores.  While  the 
work  by  Harris  seeks  to  broaden  the  scope  and  applicability  of  the  MCAS,  this  study 
explores  internal  aspects  of  the  survey.  By  separating  responses  by  demographics,  this 
thesis  will  further  explore  the  attitude  of  the  Naval  aviation  maintenance  person  with 
respect  to  safety. 

3.  Instrument  Design  and  Demographics 

The  design  of  the  MCAS  is  a  cross-sectional  one  time  look  at  a  maintenance 
organization  (see  Appendix  A).  It  is  a  self-administered  questionnaire  that  polls 
respondent  perception  about  the  safety  of  their  working  environment  at  all  levels. 
Demographic  items  preserve  anonymity  by  excluding  personal  questions  which  could  be 
linked  to  individuals  allowing  personnel  the  freedom  to  express  their  true  opinion  to  the 
perception  items  (Oneto,  1999). 

Though  there  are  many  methods  for  data  collection  from  people,  surveys,  when 
designed  properly,  are  very  effective  for  recording  respondent  scores  based  on  a 
particular  model  (Fink  &  Kosecoff,  1985).  MCAS  responses  are  forced  using  a  five  point 
Likert  scale  (see  Appendix  A).  Using  a  forced  scale  instead  of  subjective  comments 
allows  for  rapid  compilation  of  data  and  analyzation  of  responses  as  numerical  values, 
which  is  convenient  for  numerical  scoring  and  comparisons  of  scores  between 
respondents,  or  groups  of  respondents  (Fink  &  Kosecoff,  1985).  The  grouping  is 
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facilitated  by  the  six  demographic  items  of  the  survey  (Baker,  1998).  By  being  able  to 
group  types  of  maintainers  independent  of  their  scores,  it  is  possible  to  investigate  for 
unequal  perceptions  across  demographic  groups. 

The  demographic  items  in  the  MCAS  capture  aspects  of  the  maintainer  within  the 
organization  such  as  experience,  and  rank,  and  ignore  personal  information  such  as  age, 
race,  sex  and  ethnicity  (Baker,  1998).  Although  it  would  be  difficult  to  capture  every 
possible  combination  of  demographic  factors,  the  MCAS  is  able  to  capture  most  job  types 
from  most  aircraft  communities.  Each  of  the  demographic  line  items  in  the  MCAS 
represents  a  simple  factor  that  might  influence  the  scoring.  It  is  unknown  which  of  these 
demographics  constitutes  a  valid  or  invalid  factor. 

E.  SUMMARY 

HROs  are  complex  and  dynamic  by  nature  but  are  not  impervious  to  unsafe  trends 
in  human  performance.  Perrow  (1984)  points  out  that  regardless  of  an  organization’s 
structure  and  nature,  “normal  accidents”  will  continue  to  occur.  With  HROs,  these 
accidents  tend  to  occur  less  frequently,  but  the  consequences  of  the  accidents  tend  to  be 
large  in  magnitude.  These  organizations  must  put  effort  into  swimming  upstream  through 
Reason’s  safety  space  towards  increased  resistance  to  accidents.  This  happens  through 
reactive  measures  like  mishap  analysis  and  proactive  measures  to  identify  “pathogenic 
conditions”  (Reason,  1997).  Though  reactive  measures  are  in  place,  proactive  measures 
are  coming  up  to  speed. 

Two  critical  parallel  developments  in  organization  safety  theory  in  regards  to 
aviation  safety  are  the  development  of  effective  taxonomies  (e.g.,  HFACS-ME)  and  the 
identification  and  accurate  modeling  of  HROs  (e.g.,  augmented  MOSE).  The 
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development  of  CSA  and  MCAS  are  steps  to  link  the  taxonomies  and  the  organization 
models.  Since  MOSE  parallels  Reason’s  informed  culture,  it  is  possible  with  MCAS  to 
identify  those  conditions  that  are  not  conducive  to  safety  and  take  proactive  measures  to 
move  the  organization  towards  increased  resistance  to  accidents  in  the  safety  space.  Since 
3rd  MAW  is  a  HRO,  CSA  and  MCAS  allow  for  the  identification  of  the  MOSE 
components  that  require  attention. 

The  MCAS  has  been  revised  into  a  more  usable  form,  and  has  been  validated, 
showing  that  the  individual  items  do  address  specific  MOSE  components.  This  thesis  is 
another  step  into  revising  the  MCAS  further  by  looking  at  the  validity  of  the  demographic 
factors.  Mapping  the  demographic  factors  into  the  MOSE  component  scores  will  show  if 
the  factors  are  relevant  or  can  be  removed  from  the  survey.  Additionally,  if  these  factors 
do  not  account  for  the  variance  in  scores,  the  survey  is  not  asking  the  correct 
demographic  questions  and  will  require  further  revision. 
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III.  METHODOLOGY 


A.  RESEARCH  APPROACH 

The  intent  of  this  study  is  to  assess  the  maintained  s  perception  of  safety  in  his  or 
her  work  environment.  This  research  involves  the  use  of  analysis  techniques  to  partition 
the  collected  data  into  smaller  groups  based  on  demographics  then  investigate  differences 
in  responses  among  the  groups.  If  statistical  differences  are  found,  a  comparison  between 
the  group  mean  and  a  particular  squadron  mean  shows  how  a  particular  demographic 
differs  from  the  rest  of  the  squadron.  Conversely,  it  shows  which  demographics  have 
response  means  that  are  more  reflective  of  the  squadron  means. 

B.  DATA  COLLECTION 

1.  Subjects 

Surveys  are  administered  to  977  officers  and  enlisted  personnel  responsible  for 
Naval  Aviation  maintenance.  These  subjects  come  from  squadrons  and  maintenance  units 
of  the  3rd  Marine  Air  Wing  located  at  MCAS  Miramar,  CA,  Camp  Pendleton,  CA,  and 
MCAS  Yuma,  AZ.  The  aircraft  represented  are  the  AH-1  “Super  Cobra,”  UH-1  “Huey,” 
CH-53  “Super  Stallion,”  CH-46E  “Sea  Knight  ”  F/A-18D  “Night  Attack  Hornet,”  F/A-18 
“Hornet,”  AV-8B  “Harrier,”  and  the  C130T  “Hercules.” 

Additionally,  subject  matter  experts  are  interviewed  about  what  they  consider  to 
be  important  demographic  information  about  the  personnel  in  their  maintenance 
organizations.  Subject  matter  experts  are  military  aviators  with  at  least  eight  years  active 
duty  service.  The  results  of  these  interviews  are  given  in  the  next  section. 
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2.  Instrument 


The  MCAS  is  a  self-administered,  group  survey  consisting  of  two  parts:  1) 
demographics;  2)  perception.  Part  I  captures  demographic  factors  of  each  subject: 
community,  squadron,  rank,  years  of  aviation  maintenance  experience,  work  center,  and 
shift.  There  are  eight  choices  available  for  community,  with  an  additional  option  of 
“other.”  The  squadron  factor  records  the  three-digit  squadron  designator.  Embedded  in 
these  two  factors  is  aircraft  type  (seventh  factor).  Rank  is  divided  into  four  levels  of 
enlisted  personnel  and  three  levels  of  officer  personnel.  Years  of  aviation  maintenance 
experience  is  partitioned  into  seven  levels.  Work  center  or  shop,  is  divided  into  eight 
shops  with  the  option  of  “other.”  Shift  divides  subjects  into  dayshift  or  nightshift. 

Part  II  captures  subject  perception  of  his  or  her  work  environment.  There  are  42 
items,  each  of  which  is  mapped  into  a  single  augmented  MOSE  component:  process 
auditing  (six  questions),  reward  system  (eight  questions),  quality  assurance  (six 
questions),  risk  management  (nine  questions),  command  and  control  (eight  questions), 
and  communication/functional  relations  (six  questions).  Each  perception  item  asks 
subjects  to  rank  a  specific  safety  related  activity  or  aspect  of  their  organization  using  a 
five  point  Likert  rating  scale  with  verbal  anchors  as  follows:  Strongly  Disagree,  Disagree, 
Neutral,  Agree,  Strongly  Agree.  When  completed,  the  items  for  each  MOSE  component 
are  averaged  to  attain  six  composite  scores,  each  one  corresponding  to  the  subject’s  rating 
of  that  particular  augmented  MOSE  component  for  his  or  her  organization. 

3.  Procedure 

The  survey  is  administered  on  site  and  in  a  group  setting  at  the  various 
participating  Squadrons  of  the  3rd  MAW.  Additionally,  the  survey  is  given  in 
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conjunction  with  a  scheduled  maintenance  safety  presentation  on  human  factors  issues  in 
aviation.  The  Squadrons  are  in  various  stages  of  training  and  operational  tasking  at  the 
time  of  the  survey  being  administered.  The  variety  of  operational  tasking  with  which  the 
squadrons  are  simultaneously  involved  during  the  administration  of  the  MCAS  accounts 
for  much  of  the  variance  in  the  number  of  surveys  collected  from  each  squadron. 
Potential  MCAS  respondents  are  briefed  on  maintenance  issues,  the  survey  and  its 
purpose  and  questions  that  arise  pertaining  to  the  survey  are  answered  by  the  survey 
administer.  Respondents  fill  out  the  surveys  using  scannable  computer  forms.  The 
surveys  are  then  immediately  collected  upon  completion  to  allow  for  maximum 
accountability. 

For  personal  interviews,  subject  matter  experts  report  what  they  think  are  the 
important  factors  when  considering  the  demographics  of  personnel  in  their  maintenance 
organizations.  Notes  are  taken  during  these  interviews  and  a  list  is  compiled  from  the 
responses  of  the  personnel  interviewed.  This  list  represents  potential  MCAS  demographic 
items. 

C.  DATA  ANALYSIS 

1.  Data  Tabulation 

Survey  results  were  compiled  into  a  database  using  a  scanning  machine,  then 
imported  into  Microsoft  Excel.  The  spreadsheet  consists  of  rows  of  respondents  and 
columns  of  survey  items  (both  demographic  and  survey  items).  Demographic  items 
record  mainly  bivariate  and  multivariate  responses,  such  as  squadron  (aircraft  type 
embedded),  rank,  and  years  experience.  Survey  item  responses  were  assigned  a  numerical 
value  of  1  through  5  corresponding  to  the  Likert  scale,  with  higher  values  being  assigned 
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to  more  positive  responses  (strongly  agree)  and  lower  values  assigned  to  more  negative 
responses  (strongly  disagree).  Each  of  the  questions  in  the  survey  addresses  one  of  the 
six  MOSE  components.  Items  addressing  similar  MOSE  components  were  collapsed  into 
an  average  score  for  that  particular  component.  Any  items  that  were  missing  values  were 
excluded  and  not  averaged  into  the  component  score.  No  weighting  is  assigned  to  items 
in  the  event  of  a  missing  item  score.  The  three  demographic  response  items 
corresponding  to  the  respondent’s  squadron  number  were  collapsed  into  a  single  coded 
value. 

2.  Statistical  Analysis 

Microsoft  Excel  is  used  to  provide  summary  statistics  and  initial  familiarization 
with  the  data.  The  data  were  cleansed  by  removing  subject  responses  with  omitted 
demographic  items  leaving  894  responses.  Items  corresponding  to  augmented  MOSE 
components  were  averaged  for  each  respondent,  leaving  six  scores  along  with 
demographic  response  items.  If  subjects  omitted  a  perspective  item  response,  the 
component  score  is  averaged  for  the  completed  items  of  that  component.  No  weighting 
for  omitted  perspective  items  is  administered.  Histograms  of  the  demographic  make  up  of 
the  data  are  constructed.  The  data  are  then  exported  to  ARG  for  its  powerful  graphing 
capabilities.  Initial  scatterplot  matrices  of  each  component  over  all  demographic  factors 
revealed  no  linear,  exponential,  or  power  trend,  although  scatterplots  matrices  of  the 
components  over  themselves  revealed  a  linear  trend. 

The  data  are  then  transported  to  MathSoft  S-plus  for  analysis.  Categorical 
demographics  are  coded  as  factors  and  the  aov  function  is  implemented  to  fit  the  six 
component  scores  based  on  demographic  factors.  Six  models  (one  for  each  component) 
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are  constructed  for  a  regression  without  interaction  among  factors.  Six  additional  models 
are  fit  for  a  regression  looking  at  two  factor  interaction.  Each  model  is  then  simplified  by 
using  S-plus  to  remove  unimportant  terms.  Models  are  compared  and  similar  models  for 
some  of  the  components  are  found.  Three  term  interaction  models  are  not  explored  in  this 
thesis. 
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IV.  RESULTS 


A.  SIMPLE  MODELS  WITHOUT  INTERACTION 

A  scatterplot  matrix  is  constructed  to  see  if  there  are  any  trends  in  the  data.  This  is 
done  without  designating  the  independent  variables  as  factors.  The  scatterplot  shows  that 
there  is  some  relationship  between  the  component  scores,  but  little  information  about  the 
demographics  is  revealed.  Figure  3  is  a  scatterplot  matrix  for  PA  versus  each  of  the 
demographic  factors.  Visual  inspection  reveals  a  possible  relationship  between  PA  and 
Rank  and  Total  years  of  Aviation  experience. 


Figure  1.  Scatterplot  of  PA  versus  Demographic  Factors. 


Three  dimensional  bar  plots  for  the  components  versus  each  factor  are  constructed 
to  see  if  there  are  any  visual  clues  as  to  some  type  of  relationship  between  the  factors  and 
the  response.  No  linear,  log  linear  or  exponential  relationship  is  visible.  However,  the 
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plots  seem  compatible  with  those  of  normally  distributed  histograms  for  each  level  of 
factor,  all  centered  in  the  same  approximate  region  of  the  component  score  (e.g.  Figure 
4). 


Figure  2.  Three  Dimensional  Bar  Plot  of  PA  Scores.  PA  scores 
are  plotted  against  different  levels  of  Total  Years  of  Aviation 
Maintenance  Experience. 

Since  there  is  no  indication  that  a  transformation  of  the  data  is  required  due  to 
visible  trends,  linear  models  for  the  six  individual  components  are  fit  against 
untransformed  factors  without  interaction  using  the  aov  function  in  S-plus.  Model 
checking  plots  are  constructed  and  case  219  is  shown  to  have  very  high  influence  in  all 
models.  Case  219  is  an  E-6/7  with  15-20  years  of  aviation  maintenance  experience, 
works  the  day  shift  in  “other”  work  center,  in  a  VMH  squadron.  Although  there  is  no 
significance  test  associated  with  Cook’s  Distance,  case  219  is  deleted  from  the  data  set 
due  to  its  unusually  high  influence  and  new  models  were  fit  (see  Appendix  C). 

Model  checking  plots  are  constructed  to  check  the  fit  of  the  models  with  case  219 
removed.  The  scatter  plot  of  the  data  with  the  regression  superimposed  reveals  what 
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might  be  a  slight  upward  trend  in  the  response  for  all  components  (see  Appendix  D).  The 
histograms  of  the  residuals  show  that  the  distribution  of  the  residuals  appear  to  be  normal 
(see  Appendix  E),  and  the  scatterplot  of  the  residuals  versus  the  fitted  values  shows  no 
discemable  pattern  in  the  residuals  (see  Appendix  F).  The  QQ-plot  shows  that  the 
residuals  are  thin  at  both  tails  for  all  models  (see  Appendix  G),  but  the  normal  shape  is 
tenable. 


Component 

R: 2 

<7 

PA 

0.1095 

0.5529 

QA 

0.1219 

0.6327 

RS 

0.1427 

0.5896 

RM 

0.1869 

0.5690 

CC 

0.1280 

0.6349 

CR 

0.1233 

0.6846 

Table  4.  R-squared  and  o  for  Models  without  Interaction. 


Values  for  the  coefficient  of  determination,  R2,  show  that  these  models  account 
but  poorly  for  the  variance  in  the  data.  The  best  model  is  RM,  accounting  for  less  than 
19%  of  the  total  variance  (see  Table  4).  The  model  with  the  lowest  R2  is  PA  with  only 
11%  of  the  total  variance  explained.  Values  for  standard  error,  a,  indicate  that  there  is  a 
large  spread  in  the  response  values.  For  example,  with  a  perfect  R2  of  1,  the  model  for  PA 
tells  us  that  68%  of  the  respondents  score  between  3.31  and  4.42,  and  that  96%  score 
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between  2.76  and  4.97.  Since  the  only  possible  scores  are  between  1  and  5,  relatively 
high  values  for  &  are  not  much  help  with  understanding  the  data. 

The  results  from  the  analysis  of  variance  from  each  model  causes  rejection  of  the 
null  hypothesis  that  all  of  the  coefficients  in  the  model  are  zero,  accepting  the  alternate 
hypothesis  that  at  least  one  coefficient  is  not  equal  to  zero  (see  Appendix  H). 
Additionally,  the  models  for  PA,  CC  and  CR  fail  the  lack  of  fit  test  indicating  that  the 
shape  of  the  fit  is  not  correct.  These  models  do  not  do  well  in  describing  the  data. 

B.  MODELS  WITH  TWO  FACTOR  INTERACTION 

Two-factor  interaction  models  are  constructed  for  all  of  the  components  using  S- 
plus  to  see  if  more  of  the  variance  in  the  data  can  be  modeled  and  model  checking  plots 
are  constructed.  The  scatterplot  of  the  data  with  the  superimposed  regression  shows  a 
linear  trend  in  the  response  against  the  factors  (see  Appendix  I),  and  the  distribution  of 
the  residuals  appears  to  be  normal  (see  Appendix  J).  The  scatterplot  of  the  residuals 
versus  the  fitted  values  shows  no  discemable  pattern  in  the  residuals  (see  Appendix  K). 
The  QQ-plots  show  strange  behavior  at  values  close  to  zero,  but  that  they  are  close  to 
being  normal  for  all  components  (see  Appendix  L).  R2  and  &  are  given  in  Table  5. 

These  two  term  interaction  models  are  better  at  explaining  more  of  the  variance  in 
the  data  as  indicated  by  the  values  for  the  coefficients  of  determination,  however  there  is 
very  little  reduction  in  the  values  for  the  standard  error.  This  improvement  in  the  values 
for  R2  comes  at  the  cost  of  increased  complexity  in  the  models.  While  the  models  without 
interaction  have  43  terms,  the  two  term  interaction  models  have  343  terms.  By  adding 
300  terms  to  the  model,  300  degrees  of  freedom  are  lost  resulting  in  no  significant 
improvement  in  the  standard  error. 
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Component 

R2 

<7 

PA 

0.4813 

0.5246 

QA 

0.4832 

0.6034 

RS 

0.5046 

0.5572 

RM 

0.5207 

0.5431 

CC 

0.4968 

0.5996 

CR 

0.4850 

0.6523 

Table  5.  R2  and  o  for  Two  Factor  Interaction  Models. 


Analysis  of  variance  on  the  two  factor  interaction  models  causes  rejection  of  the 
null  hypothesis  that  all  of  the  coefficients  are  equal  to  zero.  The  alternative  hypothesis  is 
accepted  meaning  that  at  least  one  coefficient  is  not  equal  to  zero.  Present  in  the  anova 
tables  is  evidence  that  some  of  the  terms  are  not  necessary  in  the  model  and  that 
simplification  is  possible. 

C.  REDUCED  MODELS  WITHOUT  INTERACTION 

The  step  function  in  S-plus  is  used  to  subtract  terms  from  the  simple  models  in  an 
effort  to  simplify  the  models  without  losing  too  much  of  the  information  they  provide.  S- 
plus  accomplishes  this  by  using  Akaike’s  information  criterion  which  is  of  the  form: 

AIC  = -2logL(xm+u  ...,xn\xi,  ...,xm)  +  2r 

where  r  is  the  total  number  of  estimated  parameters.  The  AIC  is  a  value  that  penalizes  a 
model  for  having  high  complexity  when  compared  to  simpler  models  with  fewer  terms. 
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The  step  function  is  applied  to  the  simple  models  without  interaction  and  the  results  are 
given  in  Appendix  N.  The  reduced  models  have  between  15  and  33  terms  compared  to  43 
terms  for  the  original  models.  Model  checking  plots  are  constructed  with  no  significant 
graphical  differences  between  the  reduced  simple  models  and  the  simple  models  (see 
Appendices  O-R).  Values  for  the  coefficient  of  determination  and  standard  error  are 
given  in  Table  6.  As  expected,  less  of  the  total  variance  is  explained  by  the  reduced 
models,  and  the  value  for  a  increases. 


Component 

R2 

a 

PA 

0.0915 

0.5532 

QA 

0.1002 

0.6345 

RS 

0.1240 

0.5925 

RM 

0.1703 

0.5711 

CC 

0.1071 

0.6365 

CR 

0.0784 

0.6906 

Table  6.  R2  and  a  for  Reduced  Models  without  Interaction. 


D.  REDUCED  TWO  FACTOR  INTERACTION  MODELS 

The  step  function  in  S-plus  is  applied  to  the  two  factor  interaction  models  and  the 
results  are  given  in  Appendix  S.  The  data  were  fit  to  the  new  models  and  model  checking 
plots  are  constructed  and  given  in  Appendices  T-W.  The  scatterplot  of  the  data  with  the 
superimposed  regression  line  indicates  that  there  might  be  a  linear  relationship  between 
the  factors  and  the  score,  but  most  of  the  data  looks  like  a  point  cloud.  The  histograms  of 
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the  residuals  have  the  appearance  of  a  normal  distribution.  The  predicted  values  versus 
the  residuals  have  no  pattern  and  the  QQ-plots  look  normal. 

The  reduced  models  have  significantly  decreased  values  for  R2  with  little  change 
in  values  for  a.  These  models  use  between  33  and  59  terms  which  is  a  significant 
simplification  over  the  343  term  models,  but  a  lot  of  the  explanation  of  the  total  variance 
is  lost  in  the  transition.  Table  6  summarizes  R2  and  a  for  the  reduced  models. 


Component 

R2 

a 

PA 

0.1098 

0.5496 

QA 

0.1228 

0.6287 

RS 

0.1854 

0.5802 

RM 

0.2453 

0.5534 

CC 

0.1747 

0.6225 

CR 

0.1342 

0.6779 

Table  7.  R2  and  a  for  Reduced  Two  Factor  Models  with  Interaction. 

E.  COMPARING  MODELS 

The  S-plus  anova  function  is  applied  to  pairs  of  models  to  see  if  they  are 
statistically  different.  The  results  are  given  in  Appendix  U.  At  a  =  0.05,  eight  pairs  of 
models  were  found  to  be  statistically  similar,  with  the  highest  similarity  between  simple 
and  reduced  two  term  interaction  models  for  PA  and  QA  (p-Value  >  0.99).  Most  of  the 
model  pairs  are  statistically  different. 
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Choosing  the  most  appropriate  model  for  a  complex  data  set  is  compromise 
between  a  model  that  explains  enough  of  the  data  while  being  simple  enough  to  use.  The 
two  factor  interaction  models  are  too  complicated  to  be  practical  with  343  terms.  The  rest 
of  the  models  are  not  different  enough  to  distinguish  in  practice,  so  the  models  with  the 
fewest  terms  are  the  preferred  models,  which  in  this  case  are  the  reduced  simple  models. 
Realistically,  none  of  the  models  are  useful.  Not  enough  of  the  total  variance  in  the  data 
is  explained  nor  is  the  range  of  expected  scores  reduced  to  a  useful  level. 

F.  INTERVIEWS 

Subject  matter  experts  interviewed  agree  that  while  all  of  the  factors  in  the  survey 
are  important,  additional  factors  could  be  added  to  more  effectively  group  personnel  and 
provide  more  information  about  the  maintenance  organization.  They  also  think  that 
dividing  total  years  of  aviation  maintenance  experience  into  two  items,  years  worked  in 
MOS  and  years  worked  outside  of  MOS  is  necessary  to  clarify  that  Total  years  of 
aviation  maintenance  experience  is  not  simply  time  on  active  duty.  The  experts  also 
indicate  factors  that  look  at  levels  of  education  and  training  and  levels  of  morale  and 
motivation  are  important  and  should  be  included  in  the  demographic  items  of  the  survey. 
Responses  from  subject  matter  experts  are  given  in  Appendix  Y. 
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V.  CONCLUSIONS 


A.  FINDINGS 

The  results  of  this  thesis  show  that  at  the  first  level  of  interaction,  the 
demographic  factors  of  the  MCAS  poorly  account  for  the  variance  in  the  responses.  The 
models  constructed  using  linear  regression  and  analysis  of  variance  do  not  capture  the 
responses  of  the  surveyed  population,  showing  that  the  demographic  factors  have  low 
utility  in  data  analysis.  While  analysis  of  variance  shows  that  the  models  are  preferred  to 
no  model  at  all,  in  use  the  models  are  too  complex  and  do  not  provide  enough  insight  into 
the  surveyed  group. 

Since  the  component  scores  are  subjective  perceptions,  there  is  no  correct  score  to 
any  of  the  perception  items.  The  reliance  on  the  human  component  in  the  scoring  is  the 
cause  of  the  large  amount  of  variance,  and  since  variance  cannot  be  explained  by  the 
demographic  factors,  the  MCAS  appears  to  be  demographically  unbiased.  The  three 
dimensional  bar  plots  of  the  component  scores  versus  the  levels  of  factors  seem  to 
support  this  (see  Figure  4).  Either  the  MCAS  has  insignificant  biasing  across  factors  or 
the  present  factors  do  not  correctly  group  respondents  to  allow  the  biasing  to  be 
conclusively  measured. 

Although  the  demographics  do  not  effectively  group  respondents,  they  do  provide 
information  about  the  demographic  composition  of  the  surveyed  group.  This  information 
alone  can  be  useful  to  commanders  in  understanding  the  substance  of  their  squadrons.  To 
make  these  items  more  useful,  changes  to  the  MCAS  demographic  items  based  on  the 
responses  from  subject  matter  experts  are  recommended  below. 
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B.  RECOMMENDATIONS 


With  the  MCAS  moving  to  the  internet,  it  will  be  possible  to  use  more 
demographic  factors  than  what  is  currently  constrained  by  the  layout  of  scannable 
computer  response  sheets.  The  MCAS  demographic  items  should  be  changed  to  include 
the  following  items: 

1)  Check  the  box  corresponding  to  your  community: 

2)  Type  in  your  unit  number. 

3)  Type  in  the  number  of  months  have  you  been  with  your  current  squadron. 

4)  Type  in  the  total  number  of  maintenance  activities  to  which  you  have  been 

assigned. 

5)  Type  in  the  number  of  deployments  you  have  made. 

6)  Check  the  box  corresponding  to  your  rank. 

7)  Type  in  the  number  years  have  you  worked  in  your  MOS. 

8)  Type  in  the  number  of  years  have  you  worked  outside  of  your  MOS. 

9)  Type  in  any  supervisory  designations  that  you  hold. 

.  10)  Check  the  box  corresponding  to  your  work  center. 

11)  Check  the  box  corresponding  to  your  shift. 

12)  Have  you  attended  A  School? 

With  these  new  questions,  further  analysis  can  be  conducted  to  investigate  for 
valid  factors  that  properly  describe  the  data  in  the  responses,  in  addition  to  investigating 
if  personal  performance  makes  a  difference  in  scoring.  Additionally,  MCAS  could  be 
modified  to  gauge  safety  climates  in  other  military  activities  such  as  military  ordnance 


38 


handling  facilities  and  flight  deck  operations  by  adjusting  the  demographic  items  to  suit 
those  specific  activities. 
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APPENDIX  A.  43-ITEM  MAINTENANCE  CLIMATE  ASSESSMENT  SURVEY 

MAINTENANCE  CLIMATE  ASSESSMENT  SURVEY  (MCAS) 

Purpose:  The  MCAS  was  designed  to  capture  maintainer  perceptions  of  maintenance  operations  as  they 
relate  to  safety.  Your  responses  help  guide  Naval  Aviation’s  on-going  efforts  to  reduce  aviation  related 
mishaps.  Thank  you  in  advance  for  your  participation! 

Directions:  Do  not  write  on  this  form.  Fill  in  all  of  your  responses  using  the  computer  sheet  provided. 
Fill  in  each  box  that  corresponds  to  your  response  completely  using  a  pencil.  This  is  not  a  timed  event,  so 
answer  each  question  carefully  and  honestly.  Individual  responses  will  not  be  reported,  only  compiled 
results  will  be  provided  to  each  squadron. 

Part  I-  Demographics  has  six  items  requesting  unit  and  biographical  data.  This  information  will  aid  in  the 
response  analysis.  NO  attempts  will  be  made  to  identify  individuals. 

Part  II-  Perceptions  has  43  questions  pertaining  to  the  maintenance  operations.  Please  choose  the  response 
to  each  item  that  most  correctly  reflects  your  honest  opinion.  Responses  are: 

A-  Strongly  Agree  B-  Agree  C-  Neutral  D-  Disagree  E-  Strongly  Disagree 

Part  I-  Demographics 

Line  1  Fill  in  the  numbered  circle  corresponding  to  your  community? 

(l)VMGR  (2)  VMA  (3)  VMFA  (4)  HMT  (5)  HMM 

(6)  VMAQ  (7)  HMH  (8)  VMH  (9)  Other 

Line  2-4  Fill  in  the  circles  corresponding  to  your  squadron  number 

Line  5  Fill  in  the  numbered  circle  corresponding  with  you  rank 

(1)  El-3  (2)  E4-5  (3)  E6-7  (4)  E8-9  (5)W01-4  (6)01-03  (7)04-5 

Line  6  Fill  in  the  numbered  circle  corresponding  to  your  total  years 

of  Aviation  Maintenance  experience 
(1)  <1  (2)  1-2  (3)3-5  (4)6-10  (5)  1 1-15  (6)  15-20  (7)  204- 

Line  7  Fill  in  the  numbered  circle  corresponding  to  your  work  center 

(1)  Power  Plants  (2)  Airframes  (3)  Survival  (4)  Quality  Assurance 
(5)  Ordnance  (6)  Avionics  (7)  MAINT  Control  (8)  Line  (9)  Other 

Line  8  Fill  in  the  numbered  circle  corresponding  to  your  shift 

(1)  Day  (2)  Night 

Part  II  Perceptions 

Fill  in  the  lettered  circle  that  corresponds  with  your  response  to  each  item. 


SA 

A 

N 

D 

SD 

1 .  The  command  adequately  reviews  and  updates  safety. 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

2.  The  command  monitors  maintainer  qualifications  and  has 

(A) 

(B) 

(C) 

(D) 

(E) 

a  program  that  targets  training  deficiencies. 

SA 

A 

N 

D 

SD 

3.  The  command  uses  safety  and  medical  staff  to  identify/ 

(A) 

(B) 

(C) 

(D) 

(E) 
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manage  personnel  at  risk. 

4.  CDIs/QARs  routinely  monitor  maintenance  evolutions. 

5.  Tool  control  is  taken  seriously  in  the  command  and 
support  equipment  licensing  is  closely  monitored. 

6.  Signing  off  personnel  qualifications  are  taken  seriously. 

7.  Our  command  climate  promotes  safe  maintenance. 


8.  Supervisors  discourage  SOP,  NAMP,  or  other  procedural 
violations  and  encourage  reporting  safety  concerns. 

9.  Peer  influence  discourages  SOP,  NAMP,  or  other 
violations  and  individuals  feel  free  to  report  them. 

10.  Violations  of  SOP,  NAMP,  or  other  procedures  are  not 
common  in  this  command. 

11.  The  command  recognizes  individual  safety  achievement 
through  rewards  and  incentives. 

12.  Personnel  are  comfortable  approaching  supervisors 
about  personal  problems/illness 

13.  Safety  NCO,  QAR,  and  CDI,  are  sought  after  billets. 


14.  Unprofessional  behavior  is  not  tolerated  in  the  command 


15.  The  command  has  a  reputation  for  quality  maintenance 
and  sets  standards  to  maintain  quality  control. 

16.  QA  and  Safety  are  well  respected,  and  are  seen  as 
essential  to  mission  accomplishment. 

17.  QARs/CDIs  sign-off  after  required  actions  are  complete 
and  are  not  pressured  by  supervisors  to  sign-off. 

1 8.  Maintenance  on  detachments  is  of  the  same  quality 
as  that  at  home  station. 

19.  Required  publications/tools/equipment  are  available, 
current/serviceable,  and  used. 

20.  QARs  are  helpful,  and  QA  is  not  “feared”  in  my  unit. 


21.  Multiple  job  assignments  and  collateral  duties  adversely 
affect  maintenance. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 
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22.  Safety  is  part  of  maintenance  planning,  and  additional 
training/support  is  provided  as  needed. 

23.  Supervisors  recognize  unsafe  conditions  and  manage 
hazards  associated  with  maintenance  and  the  flight-line. 

24.  I  am  provided  adequate  resources,  time,  personnel  to 
accomplish  my  job. 

25.  Personnel  turnover  does  not  negatively  impact  the 
command’s  ability  to  operate  safely. 

26.  Supervisors  are  more  concerned  with  safe  maintenance 
than  the  flight  schedule,  and  do  no  permit  cutting  corners. 

27.  Day/Night  Check  have  equal  workloads,  and  staffing  is 
sufficient  on  each  shift. 

28.  Supervisors  shield  personnel  from  outside  pressures  and 
are  aware  of  individual  workload. 

29.  Based  upon  my  command’s  current  assets/manning  it 
is  not  over-committed. 

30.  My  command  temporarily  restricts  maintainers  who  are 
having  a  problems. 

31.  Safety  decisions  are  made  at  the  proper  levels  and 
work  center  supervisor  decisions  are  respected. 

32.  Supervisors  communicate  command  safety  goals  and 
are  actively  engaged  in  the  safety  program. 

33.  Supervisors  set  the  example  for  following  to 
maintenance  standards  and  ensure  compliance. 

34.  In  my  command  safety  is  a  key  part  of  all  maintenance 
operations,  and  all  are  responsible/accountable  for  safety. 

35.  Safety  education  and  training  are  comprehensive  and 
effective. 

36.  All  maintenance  evolutions  are  properly  briefed, 
supervised,  and  staffed  by  qualified  personnel. 

37.  Maintenance  Control  is  effective  in  managing  all 
maintenance  activities. 

38.  Good  communication  exists  up/down  the  chain  of 
command. 

39.  I  get  all  the  information  I  need  to  do  my  job  safely. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 
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40.  Work  center  supervisors  coordinate  their  actions. 


41 .  My  command  has  effective  pass-down  between  shifts. 

42.  Maintenance  Control  troubleshoots/resolves  gripes  before 
flight. 

43.  Maintainers  are  briefed  on  potential  hazards  associated 
with  maintenance  activities. 


SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 

SA 

A 

N 

D 

SD 

(A) 

(B) 

(C) 

(D) 

(E) 
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APPENDIX  B.  MODEL  OF  SAFETY  EFFECTIVENESS  COMPONENTS. 


COMPONENT  1:  PROCESS  AUDITING 

1 .  The  command  adequately  reviews  and  updates  safety  practices. 

2.  The  command  monitors  maintainer  qualifications  and  has  a  program  that  targets 
training  deficiencies. 

3.  The  command  uses  safety  and  medical  staff  to  identify/manage  personnel  at  risk. 

4.  CDIs/QARs  routinely  monitor  maintenance  evolutions. 

5.  Tool  Control  is  taken  seriously  in  the  command  and  support  equipment  licensing  is 
closely  monitored. 

6.  Signing  personal  qualifications  are  taken  seriously. 

COMPONENT  2:  Reward  System  and  Safety  Climate 

1 .  Our  command  climate  promotes  safe  maintenance  and  flight  operations. 

2.  Supervisors  discourage  SOP,  NAMP  or  other  procedure  violations  and  encourage 
reporting  safety  concern. 

3.  Peer  influence  discourages  SOP,  NAMP  or  other  violations  and  individuals  feel  free 
to  report  them. 

4.  Violations  of  SOP,  NAMP  or  other  procedures  are  not  common  in  this  command. 

5.  The  command  recognizes  individual  safety  achievement  through  rewards  and 
incentives. 

6.  Personnel  are  comfortable  approaching  supervisors  about  personal  problems/illness. 

7.  Safety  NCO,  QAR,  and  CDI,  are  sought  after  billets. 

8.  Unprofessional  behavior  is  not  tolerated  in  the  command. 

COMPONENT  3:  QUALITY  ASSURANCE 

1.  The  command  has  a  reputation  for  quality  maintenance  and  has  set  standards  to 
maintain  quality  control. 

2.  QA  and  Safety  are  well  respected,  and  are  seen  as  essential  to  mission 
accomplishment. 

3.  QARs/CDIs  sign-off  after  required  actions  are  complete  and  are  not  pressured  by 
supervisors  to  sign-off. 

4.  Maintenance  on  detachments  is  the  same  quality  as  that  at  home  station. 

5.  Required  publications/tools/equipment  are  available,  current/serviceable,  and  used. 

6.  QARs  are  helpful,  and  QA  is  not  “feared”  in  my  unit. 
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COMPONENT  4:  RISK  MANAGEMENT 


1.  Multiple  job  assignments  and  collateral  duties  adversely  affect  maintenance. 

2.  Safety  is  part  of  maintenance  planning,  and  additional  training/support  is  provided  as 
needed. 

3.  Supervisors  recognize  unsafe  conditions  and  manage  hazards  associated  with 
maintenance  and  the  flight  line. 

4.  I  am  provided  adequate  resources,  time,  personnel  to  accomplish  my  job. 

5.  Personnel  turnover  does  not  negatively  impact  the  command’s  ability  to  operate 
safely. 

6.  Supervisors  are  more  concerned  with  safe  maintenance  than  the  flight  schedule,  and 
do  not  permit  cutting  comers. 

7.  Day/Night  Check  have  equal  workloads,  and  staffing  is  sufficient  on  each  shift. 

8.  Supervisors  shield  personnel  from  outside  pressures  and  are  aware  of  individual 
workload. 

9.  Based  upon  my  command’s  current  assets/manning  it  is  not  over-committed. 

COMPONENT  5:  COMMAND  AND  CONTROL 

1 .  My  command  temporarily  restricts  maintainers  who  are  having  problems. 

2.  Safety  decisions  are  made  at  the  proper  levels,  work  center  supervisors  decisions  are 
respected. 

3.  Supervisors  communicate  command  safety  goals  and  are  actively  engaged  in  the 
safety  program. 

4.  Supervisors  set  the  example  for  following  to  maintenance  standards  and  ensure 
compliance. 

5.  In  my  command  safety  is  a  key  part  of  all  maintenance  operations  and  all  are 
responsible/accountable  for  safety. 

6.  Safety  education  and  training  are  comprehensive  and  effective. 

7.  All  maintenance  evolutions  are  properly  briefed,  supervised,  and  staffed  by  qualified 
personnel. 

8.  Maintenance  control  is  effective  in  managing  all  maintenance  activities. 


COMPONENT  6:  COMMUNICATION  /  FUNCTIONAL  RELATIONSHIPS 

1.  Good  communication  exists  up/down  the  chain  of  command. 

2.  I  get  all  the  information  I  need  to  do  my  job  safely. 

3.  Work  center  supervisors  coordinate  their  actions. 

4.  My  command  has  effective  pass-down  between  shifts. 

5.  Maintenance  Control  troubleshoots/resolves  gripes  before  flight. 

6.  Maintainers  are  briefed  on  potential  hazards  associated  with  maintenance  activities. 
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APPENDIX  D.  SIMPLE  MODELS:  SCATTERPLOTS  WITH  SUPERIMPOSED 
REGRESSION  LINE 


PA 


RS 


Fitted  :  {lactor(Community)  +  lactor(Squadron)  +  factor(Rank)  +  }actor{Tot.Years)  +  factor(Shop)  +  factorfSh  Fitted  :  (factor(Community)  +  (aclor(Squadron)  +  factor(Rank)  +  )actor(Tot. Years)  ♦  tactor(Shop)  +  Jador(Sh 


QA  RM 


Fitted  :  (faetor(Community)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot.Years)  +  factor(Shop)  +  factor(Sh  Fitted  :  (factor(Community)  +  tactor(Squadron)  +  factor(Rank)  +  tactor{Tol.Years)  +  factor(Shop)  +  factor(Sh 


CC  CR 


Fitted :  (factor(Community)  +  tactor(Squadron)  +  factor  (Rank)  +  factor(Tot.  Years)  +  factor(Shop)  +  tactor(Sh  Fitted  :  (factor(Community)  +  (actor(Squadron)  +  factor(Rank)  +  lactor(Tot.Years)  +  factor(Shop)  +  factor(Sh' 
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APPENDIX  E.  SIMPLE  MODELS:  HISTOGRAMS  OF  RESIDUALS 
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APPENDIX  H.  SIMPLE  MODELS:  ANALYSIS  OF  VARIANCE 


PA 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  8.3925  1.198929  3.922101  0.0003198 

factor (Squadron)  13  8.0763  0.621257  2.032343  0.0160499 

factor (Rank)  6  6.0359  1.005978  3.290893  0.0033082 

factor (Tot. Years)  6  3.4403  0.573388  1.875746  0.0821892 

factor (Shop)  9  3.2828  0.364753  1.193231  0.2957039 

factor (Shift)  1  2.7153  2.715335  8.882780  0.0029608 

Residuals  850  259.8325  0.305685 

QA 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  8.3910  1.198710  2.994574  0.0041284 

factor (Squadron)  13  14.3600  1.104618  2.759519  0.0007443 

factor (Rank)  6  12.2469  2.041156  5.099143  0.0000372 

factor (Tot. Years)  6  7.6351  1.272514  3.178950  0.0043234 

factor (Shop)  9  3.9649  0.440543  1.100548  0.3596172 

factor ( Shift)  1  0.6428  0.642779  1.605767  0.2054349 

Residuals  850  340.2498  0.400294 

RS 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  3.7699  0.538562  1.549360  0.1471909 

factor (Squadron)  13  12.7604  0.981569  2.823824  0.0005582 

factor (Rank)  6  17.8319  2.971978  8.549925  0.0000000 

factor (Tot. Years)  6  7.6649  1.277490  3.675142  0.0013048 

factor (Shop)  9  5.4436  0.604846  1.740050  0.0761483 

factor (Shift)  1  1.7291  1.729082  4.974302  0.0259869 

Residuals  850  295.4624  0.347603 

RM 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  15.9912  2.284458  7.056521  0.0000000 

factor (Squadron)  13  11.8830  0.914080  2.823525  0.0005590 

factor (Rank)  6  19.2158  3.202630  9.892685  0.0000000 

factor (Tot. Years)  6  10.5421  1.757022  5.427311  0.0000161 

factor (Shop)  9  4.8880  0.543110  1.677627  0.0901813 

factor (Shift)  1  0.7475  0.747503  2.308980  0.1290003 

Residuals  850  275.1766  0.323737 

cc 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  9.8503  1.407189  3.490669  0.0010676 

factor (Squadron)  13  12.2826  0.944816  2.343707  0.0045142 

factor (Rank)  6  15.1026  2.517097  6.243904  0.0000020 

factor (Tot. Years)  6  6.7176  1.119607  2.777295  0.0111256 

factor (Shop)  9  3.5477  0.394188  0.977822  0.4567736 

factor (Shift)  1  2.8159  2.815870  6.985039  0.0083707 

Residuals  850  342.6594  0.403129 


Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  12.4423  1.777478  3.792880  0.0004599 

factor (Squadron)  13  11.0908  0.853140  1.820477  0.0361248 

factor (Rank)  6  18.6478  3.107964  6.631942  0.0000007 

factor (Tot. Years)  6  9.4231  1.570522  3.351265  0.0028616 

factor (Shop)  9  3.7036  0.411516  0.878115  0.5443707 

factor (Shift)  1  0.7389  0.738887  1.576677  0.2095850 

Residuals  850  398.3402  0.468636 
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APPENDIX  I. 


TWO  FACTOR  INTERACTION  MODELS:  SCATTERPLOTS 
WITH  SUPERIMPOSED  REGRESSION  LINE 


Fitted  :  (factor(Commumty)  +  tactor(Squadron)  +  factor(Rank)  +  factor(Tot. Years)  +  tactor(Shop)  +  factor{Shil  Fitted  :  (factor (Comm unity}  +  tactor(Squadron)  +  factor(Rank)  +  factor(Tot.Years)  +  factor(Shop)  +  factor(Shrt 
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Fitted  :  (factor(Community)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot.Years)  +  factor(Shop)  faetor($hh  Fitted  :  (faetor(Community)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot.  Years)  +  factor(Shop)  +  tactor(Shil 
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Fitted  :  (tactor(Community)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot. Years)  +  factor(Shop)  +  factor(Shi)  Fitted :  (factor(Community)  +  factor(Squadron)  +  factor(Rank)  +  1actor{Tot. Years)  +  factor(Shop)  *■  Jactor(Shil 
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APPENDIX  J.  TWO  FACTOR  INTERACTION  MODELS:  HISTOGRAMS  OF 
RESIDUALS 

PA  QA 


-3  -2  -10  12  -2-1012 

resid(cc.2.aov)  resid(cr.2.aov) 
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APPENDIX  K.  TWO  FACTOR  INTERACTION  MODELS:  RESIDUALS 
VERSUS  FITTED  VALUES  WITH  LOESS  SMOOTHING 


QA 
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APPENDIX  M.  TWO  FACTOR  INTERACTION  MODELS:  ANALYSIS  OF 
VARIANCE 

PA 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  8.3925  1.198929  4.356653  0.0001027 

factor (Squadron)  13  8.0763  0.621257  2.257518  0.0068172 

factor (Rank)  6  6.0359  1.005978  3.655510  0.0014394 

factor (Tot. Years)  6  3.4403  0.573388  2.083571  0.0534953 

factor (Shop)  9  3.2828  0.364753  1.325436  0.2203764 

factor (Shift)  1  2.7153  2.715335  9.866954  0.0017732 

factor ( Community) : factor (Rank)  27  8.5470  0.316557  1.150301  0.2755360 

factor (Community) : factor (Tot. Years)  32  5.5049  0.172029  0.625117  0.9480617 

factor (Community) : factor (Shop)  46  16.8120  0.365478  1.328069  0.0778513 

factor (Community) : factor (Shift)  5  4.8802  0.976038  3.546715  0.0036412 

factor (Squadron) : factor (Rank)  27  7.1875  0.266205  0.967334  0.5134968 

factor (Squadron) ; factor (Tot. Years)  37  16.5409  0.447050  1.624486  0.0126771 

factor (Squadron) : factor (Shop)  56  23.4663  0.419042  1.522709  0.0108588 

factor (Squadron) : factor (Shift)  6  3.7496  0.624928  2.270855  0.0356434 

factor (Rank) : factor (Tot. Years)  8  3.4152  0.426899  1.551259  0.1367135 

factor (Rank) : factor (Shop)  15  5.1111  0.340743  1.238189  0.2382294 

factor (Rank) : factor (Shift)  2  0.4713  0.235625  0.856213  0.4253331 

factor (Tot. Years) : factor (Shop)  29  8.8454  0.305014  1.108357  0,3199687 

factor (Tot. Years) : factor (Shift)  3  0.7165  0.238846  0.867915  0.4574968 

factor (Shop) : factor (Shift)  7  3.2273  0.461047  1.675347  0.1124277 

Residuals  550  151.3572  0.275195 

QA 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  8.3910  1.198710  3.292186  0.0019394 

factor (Squadron)  13  14.3600  1.104618  3.033770  0.0002431 

factor (Rank)  6  12.2469  2.041156  5.605916  0.0000116 

factor (Tot. Years)  6  7.6351  1.272514  3.494886  0.0021165 

factor (Shop)  9  3.9649  0.440543  1.209925  0.2860380 

factor (Shift)  1  0.6428  0.642779  1.765354  0.1845090 

factor (Community) : factor (Rank)  27  10.0763  0.373196  1.024962  0.4313503 

factor (Community) : factor (Tot. Years)  32  10.2932  0.321663  0.883428  0.6538426 

factor (Community) : factor (Shop)  46  23.9700  0.521087  1.431136  0.0362521 

factor (Community) : factor (Shift)  5  11.2388  2.247751  6.173318  0.0000140 

factor (Squadron) : factor (Rank)  27  17.7225  0.656389  1.802734  0.0083195 

f actor (Squadron) : factor (Tot. Years)  37  15.4891  0,418624  1.149727  0.2540719 

factor (Squadron) : factor (Shop)  56  22.7429  0.406123  1.115393  0.2703154 

factor (Squadron) : factor (Shift)  6  3.5604  0.593400  1.629737  0.1365990 

factor (Rank) : factor (Tot. Years)  8  5.2739  0.659237  1.810557  0.0725066 

factor ( Rank) : factor (Shop)  15  7.4882  0.499213  1.371060  0.1561216 

factor (Rank) : factor (Shift)  2  1.7580  0.879015  2.414162  0.0903895 

factor (Tot. Years) : factor (Shop)  29  7.0553  0.243286  0.668170  0.9075797 

factor (Tot. Years) : factor (Shift)  3  0.8364  0.278789  0.765679  0.5136070 

factor (Shop) : factor (Shift)  7  2.4857  0.355105  0.975276  0.4483208 

Residuals  550  200.2591  0.364107 

RS 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

factor (Community)  7  3.7699  0.538562  1.734800  0.0984123 

factor (Squadron)  13  12.7604  0.981569  3.161802  0.0001365 

factor (Rank)  6  17.8319  2.971978  9.573249  0.0000000 

factor (Tot. Years)  6  7.6649  1.277490  4.115013  0.0004719 

factor (Shop)  9  5.4436  0.604846  1.948314  0.0432207 

factor (Shift)  1  1.7291  1.729082  5.569667  0.0186225 

factor (Community) : factor (Rank)  27  8.6010  0.318554  1.026117  0.4297560 

factor (Community) : factor (Tot. Years)  32  5.9900  0.187188  0.602964  0.9597519 

factor (Community) : factor (Shop)  46  19.4596  0.423035  1.362667  0.0607806 

factor (Community) : factor (Shift)  5  6.8711  1.374220  4.426596  0.0005820 

factor (Squadron) : factor (Rank)  27  12.1684  0.450680  1.451719  0.0672413 

factor (Squadron) : factor (Tot. Years)  37  14.7707  0.399208  1.285918  0.1242051 

factor (Squadron) : factor (Shop)  56  26.1193  0.466416  1.502405  0.0132179 

factor (Squadron) : factor (Shift)  6  4.7507  0.791781  2.550461  0.0191331 

factor (Rank) : factor (Tot. Years)  8  3.5565  0.444564  1.432017  0.1800429 

factor (Rank) : factor (Shop)  15  4.1334  0.275561  0.887628  0.5783215 

factor (Rank) : factor (Shift)  2  1.6918  0.845879  2.724720  0.0664497 

factor (Tot. Years) : factor (Shop)  29  10.6388  0.366855  1.181702  0.2373524 

factor(Tot .Years) : factor (Shift)  3  2.3787  0.792884  2.554013  0.0546501 

factor (Shop) : factor (Shift)  7  3.5872  0.512457  1.650712  0.1187383 

Residuals  550  170.7454  0.310446 
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RM 


Df 

factor (Community)  7 
factor (Squadron)  13 
factor (Rank)  6 
factor (Tot . Years)  6 
factor (Shop)  9 
factor (Shift)  1 
factor (Community) : factor (Rank)  27 
factor (Community) : factor (Tot . Years)  32 
factor (Community) : factor (Shop)  46 
factor (Community) : factor (Shift )  5 

factor (Squadron) : factor (Rank)  27 
factor (Squadron) : factor (Tot . Years)  37 
factor (Squadron) : factor (Shop)  56 
factor (Squadron) : factor (Shift )  6 

factor (Rank) : factor (Tot. Years)  8 
factor (Rank) : factor (Shop)  15 
factor (Rank) : factor (Shift)  2 
factor (Tot .Years) : factor (Shop)  29 
factor (Tot .Years) : factor (Shift )  3 

factor (Shot) : factor (Shift)  7 
Residuals  550 

CR 

Df 

factor (Community)  7 
factor (Squadron)  13 
factor (Rank)  6 
factor (Tot. Years)  6 

factor (Shop)  9 
factor (Shift)  1 
factor (Community) : factor (Rank)  27 
factor (Community) : factor (Tot . Years)  32 
factor (Community) : factor (Shop)  46 
factor (Community) : factor (Shift)  5 
factor (Squadron) : factor (Rank)  27 
factor (Squadron) : factor (Tot . Years)  37 
factor (Squadron) : factor (Shop)  56 
factor (Squadron) : factor (Shift)  6 
factor (Rank) : factor (Tot . Years)  8 
factor (Rank) : factor (Shop)  15 
factor (Rank) : factor (Shift )  2 

factor (Tot . Years) : factor (Shop)  29 
factor (Tot .Years) : factor (Shift)  3 
factor (Shop) : factor (Shift)  7 
Residuals  550 

cc 

Df 

factor (Community)  7 
factor (Squadron)  13 
factor (Rank)  6 
factor (Tot .Years)  6 
factor (Shop)  9 
factor (Shift)  1 
factor (Community) : factor (Rank)  27 
factor (Community) : factor (Tot .Years)  32 
factor (Community) : factor (Shop)  46 
factor (Community) : factor (Shift)  5 
factor (Squadron) : factor (Rank)  27 
factor (Squadron) : factor (Tot . Years)  37 
factor (Squadron) : factor (Shop)  56 
factor (Squadron) : factor (Shift)  6 
factor (Rank) : factor (Tot . Years)  8 

factor (Rank) : factor (Shop)  15 
factor (Rank) : factor (Shift)  2 
factor (Tot .Years) : factor (Shop)  29 
factor (Tot .Years) : factor (Shift)  3 

factor (Shop) : factor (Shift)  7 

Residuals  550 


Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 
15.9912  2.284458  7.74590  0.0000000 

11.8830  0.914080  3.09937  0.0001810 

19.2158  3.202630  10.85914  0.0000000 
10.5421  1.757022  5.95752  0.0000048 

4.8880  0.543110  1.84152  0.0583837 

0.7475  0.747503  2.53455  0.1119530 

6.1172  0.226562  0.76820  0.7946819 

7.0593  0.220602  0.74799  0.8421223 

15.9480  0.346696  1.17554  0.2054702 

12.6939  2.538785  8.60824  0.0000001 

10.8087  0.400322  1.35737  0.1093488 

12.4608  0.336778  1.14191  0.2636165 

20.8355  0.372062  1.26155  0.1036550 

5.6544  0.942395  3.19537  0.0043116 

2.4103  0.301291  1.02158  0.4183172 

4.4618  0.297452  1.00857  0.4442584 

2.6308  1.315404  4.46013  0.0119820 

8.4000  0.289655  0.98213  0.4942530 

1.2055  0.401827  1.36247  0.2533738 

2.2818  0.325976  1.10528  0.3581240 

162.2087  0.294925 


Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 
12.4423  1.777478  4.177394  0.0001697 
11.0908  0.853140  2.005033  0.0185347 
18.6478  3.107964  7.304274  0.0000002 
9.4231  1.570522  3.691009  0.0013214 
3.7036  0.411516  0.967136  0.4662855 
0.7389  0.738887  1.736518  0.1881293 
10.0529  0.372329  0.875041  0.6494108 
10.0165  0.313016  0.735645  0.8559696 
30.3247  0.659233  1.549317  0.0137593 
8.1375  1.627495  3.824905  0.0020492 
14.7948  0.547957  1.287797  0.1526769 
19.5299  0.527835  1.240507  0.1600404 
36.1636  0.645778  1.517696  0.0114023 
3.4718  0.578626  1.359876  0.2288249 
4.5586  0.569829  1.339200  0.2212122 
6.5391  0.435941  1.024540  0.4275257 
1.2651  0.632559  1.486627  0.2270415 
14.0630  0.484930  1.139672  0.2827731 
2.2102  0.736717  1.731418  0.1594515 
3.1879  0.455411  1.070298  0.3811256 
234.0246  0.425499 


Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 
9.8503  1.407189  3.913888  0.0003535 
12.2826  0.944816  2.627865  0.0014467 
15.1026  2.517097  7.000932  0.0000003 
6.7176  1.119607  3.114022  0.0052210 
3.5477  0.394188  1.096376  0.3634308 
2.8159  2.815870  7.831926  0.0053132 
8.8346  0.327206  0.910074  0.5979349 
8.7848  0.274526  0.763554  0.8237228 
20.7928  0.452016  1.257217  0.1253496 
10.1673  2.033454  5.655750  0.0000426 
11.6354  0.430941  1.198599  0.2264270 
17.7331  0.479274  1.333030  0.0940917 
27.8577  0.497460  1.383610  0.0390184 
5.1754  0.862573  2.399120  0.0268520 
5.4297  0.678707  1.887723  0.0595526 
6.9073  0.460485  1.280770  0.2089887 
1.7807  0.890356  2.476393  0.0849826 
14.6705  0.505881  1.407033  0.0788824 
2.0737  0.691223  1.922534  0.1248178 
3.0709  0.438695  1.220166  0.2894603 
197.7456  0.359537 
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APPENDIX  N.  REDUCED  SIMPLE  MODELS 


Model : 

PA  -  factor (Squadron)  +  factor (Tot .Years)  +  factor (Shift ) 


scale:  0.3056853 

Sum  of  Sq  RSS  Cp 

265.0655  281.5725 
19.09854  284.1640  289.0550 
6.98011  272.0456  284.8844 
2.98713  268.0526  283.9482 


Df 

<none> 

factor ( Squadron)  1 9 
factor (Tot .Years)  6 
factor (Shift)  1 


Terms : 

Sum  of  Squares 
Deg.  of  Freedom 


factor (Squadron) 
16.4369 
19 


factor (Tot .Years) 
7.2861 
6 


factor (Shift)  Residuals 
2.9871  265.0655 

1  866 


Residual  standard  error:  0.5532452 
Estimated  effects  may  be  unbalanced 


QA  ~  factor (Squadron)  +  factor (Tot. Years)  +  factor (Shift) 


scale:  0.4002939 


Df 

<none> 

factor (Squadron)  19 
factor (Tot .Years)  6 
factor (Shift)  1 


Sum  of  Sq  RSS  Cp 

348.6818  370.2977 
23.06513  371.7469  378.1516 
15.45358  364.1354  380.9477 
0.90043  349.5822  370.3975 


Terms : 


factor (Squadron) 
Sum  of  Squares  22.4130 

Deg.  of  Freedom  19 


factor (Tot .Years) 
15.4952 
6 


factor (Shift)  Residuals 
0.9004  348.6818 

1  866 


Residual  standard  error:  0.6345352 
Estimated  effects  may  be  unbalanced 


Model : 

RS  ~  factor (Squadron)  +  factor (Rank)  +  factor (Tot .Years)  +  factor (Shift) 


scale:  0.3476029 


Df 

<none> 

factor (Squadron)  19 
factor (Rank)  6 
factor (Tot .Years)  6 
factor (Shift)  1 


Sum  of  Sq  RSS  Cp 

301.9366  324.8784 
16.56125  318.4979  328.2308 
5.50006  307.4367  326.2072 
7.70716  309.6438  328.4143 
1.04953  302.9861  325.2327 


Terms : 


factor (Squadron) 
Sum  of  Squares  16.0094 

Deg.  of  Freedom  19 


factor (Rank) 
18.0278 
6 


factor (Tot .Years) 
7.6388 
6 


factor (Shift)  Residuals 
1.0495  301.9366 

1  860 


Residual  standard  error:  0.5925277 
Estimated  effects  may  be  unbalanced 


Model : 

RM  -  factor (Squadron)  +  factor (Rank)  +  factor (Tot .Years) 


scale:  0.3237372 


<none> 


Df  Sum  of  Sq  RSS  Cp 

280.8129  301.5321 
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factor (Squadron)  19  23.32159  304.1345  312.5517 
factor (Rank)  6  5.07801  285.8909  302.7253 

factor (Tot. Years)  6  10.54164  291.3546  308.1889 


Terms : 

Sum  of  Squares 
Deg.  of  Freedom 


factor { Squadron) 
27.8628 
19 


factor (Rank) 
19.2269 
6 


factor (Tot .Years) 
10.5416 
6 


Residuals 

280.8129 

861 


Residual  standard  error:  0.5710932 
Estimated  effects  may  be  unbalanced 


Model : 

CC  -  factor (Squadron)  +  factor (Tot .Years)  +  factor (Shift) 


scale:  0.4031287 


Df  Sum  of  Sq  RSS  Cp 

<none>  350.8909  372.6598 

factor (Squadron)  19  22.16263  373.0535  379.5036 

factor (Tot. Years)  6  17.43463  368.3255  385.2569 

factor (Shift)  1  2.80445  353.6953  374.6580 


Terms : 

Sum  of  Squares 
Deg.  of  Freedom 


factor (Squadron) 
22.0953 
19 


factor (Tot. Years) 
17.1855 
6 


factor (Shift) 
2.8045 
1 


Residuals 

350.8909 

866 


Residual  standard  error:  0.636542 
Estimated  effects  may  be  unbalanced 


Model : 

CR  ~  factor (Community)  +  factor (Tot .Years)  +  factor (Shift) 


scale:  0.4686355 


Df  Sum  of  Sq  RSS  Cp 

<none>  418.7697  432.8288 

factor (Community)  7  10.24208  429.0118  436.5100 

factor (Tot. Years)  6  21.33850  440.1082  448.5437 

factor (Shift)  1  1.89642  420.6661  433.7879 


Terms : 

Sum  of  Squares 
Deg.  of  Freedom 


factor (Community) 
12.4423 
7 


factor (Tot .Years) 
21.2783 
6 


factor (Shift) 
1.8964 
1 


Residuals 

418.7697 

878 


Residual  standard  error:  0.690622 
Estimated  effects  may  be  unbalanced 
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APPENDIX  O.  REDUCED  SIMPLE  MODELS:  SCATTERPLOTS  WITH 
SUPERIMPOSED  REGRESSION  LINE 


Fitted  :  factor(Squadron)  factor{Tot. Years)  +  tactor(Shitt) 


Fitted  :  fador(Community)  +  factor(Tot.Years)  +  factor{Shift) 
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APPENDIX  P.  REDUCED  SIMPLE  MODELS:  HISTOGRAMS  OF  RESIDUALS 
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APPENDIX  O.  REDUCED  SIMPLE  MODELS:  RESIDUALS  VERSUS  FITTED 


VALUES 


3.4  3.6  3.8  4.0  4.2  4.4 


Fitted :  lactor(Squadron)  +  factorfTot.  Years)  +  factor(Shift) 


3.0  3.2  3.4  3.6  3.8  4,0  4.2 

Fitted :  factor(Squadron)  +  faetor(Rank)  +  factor(Tot.Years)  +  factor(Shift) 


cc 


3.2  3.4  3.6  3.8  4.0  4.2 

Fitted  :  faetor(Squadron)  +  factor(Tol.Years)  +  factor(Shift) 


3.2  3.4  3.6  3.8  4.0 

Fitted  :  factor(Community)  +  factorfTot.Years)  +  factor(Shift) 
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APPENDIX  R.  REDUCED  SIMPLE  MODELS:  QQ-PLOTS 


of  Standard  Normal 


Quantiles  of  Standard  Normal 
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APPENDIX  S.  REDUCED  TWO  FACTOR  INTERACTION  MODELS 


Model : 

PA  -  factor (Community)  +  factor (Squadron)  +  factor (Tot .Years)  +  factor (Shift )  + 
factor (Community) : factor (Shift) 

scale:  0.2751949 


Df 

<none> 

factor (Squadron)  12 
factor (Tot .Years)  6 
factor (Community) : factor (Shift)  5 


Sum  of  Sq  RSS 

259.7296 
8.235337  267.9649 
7.737269  267.4669 
5.319236  265.0488 


Cp 

277.8925 

279.5231 

282.3274 

280.4597 


Model : 

QA  -  factor (Community)  +  factor (Squadron)  +  factor (Tot .Years)  +  factor (Shift)  + 
factor (Community) : factor (Shift) 


scale:  0.3641075 


<none> 
factor (Squadron) 
factor (Tot .Years) 
factor (Community) : factor (Shift) 


Df  Sum  of  Sq  RSS  Cp 

339.8904  363.9215 
12  12.36975  352.2601  367.5526 

6  16.34776  356.2381  375.8999 

5  8.42356  348.3139  368.7039 


Model : 

RS  -  factor (Squadron)  +  factor (Rank)  +  factor (Tot .Years)  +  factor (Shop)  +  factor (Shift )  + 
factor (Squadron) : factor (Shift)  +  factor (Tot .Years) : factor (Shift) 

scale:  0.3104461 


<none> 
factor (Rank) 
factor (Shop) 
factor (Squadron) : factor (Shift) 
factor (Tot. Years) : factor (Shift) 


Df  Sum  of  Sq  RSS 

280.7776 
6  5.38955  286.1672 

9  6.45093  287.2286 

12  11.55724  292.3349 

5  4.31919  285.0968 


Cp 

317.4103 

319.0745 

318.2732 

321.5168 

318.6250 


Model : 

RM  -  factor (Squadron)  +  factor (Rank)  +  factor (Tot .Years)  +  factor (Shop)  +  factor (Shift )  + 
factor (Squadron) : factor (Shift)  +  factor (Tot .Years) : factor (Shift) 

scale:  0.2949249 


<none> 
factor (Rank) 
factor (Shop) 
factor (Squadron) : factor (Shift) 
factor (Tot .Years) : factor (Shift) 


Df  Sum  of  Sq  RSS 

255.4226 
6  3.99087  259.4134 

9  5.55127  260.9738 

12  17.43743  272.8600 

5  3.57653  258.9991 


Cp 

290.2237 

290.6755 

290.4663 

300.5829 

290.8510 


Model : 

CC  -  factor (Community)  +  factor (Squadron)  +  factor (Rank)  +  factor (Tot .Years)  + 
factor (Shift)  +  factor (Community) : factor (Shift)  +  factor (Squadron) : factor (Shift )  + 

factor (Rank) : factor (Shift)  +  factor (Tot .Years) : factor (Shift) 

scale:  0.3595374 


<none> 

factor (Community) : factor (Shift ) 
factor (Squadron) : factor (Shift ) 
factor (Rank) : factor (Shift) 
factor (Tot .Years) : factor (Shift) 


Df  Sum  of  Sq  RSS 

324.3285 
0  0.000000  324.3285 

7  5.543296  329.8718 

5  4.227122  328.5556 

5  5.275932  329.6045 


Cp 

364.5967 

364.5967 

365.1065 

365.2285 

366.2773 
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Model : 

CR  ~  factor (Community)  +  factor (Rank)  +  factor (Tot .Years )  +  factor (Shift)  + 
factor (Community) : factor (Shift)  +  factor (Rank) : factor (Shift )  +  factor (Tot .Years) 
factor (Shift) 

scale:  0.4254993 


<none> 

factor (Community) : factor (Shift ) 
factor (Rank) : factor (Shift) 
factor (Tot .Years) : factor (Shift) 


Df  Sum  of  Sq 

6  9.058145 
5  7.047982 
5  7.483000 


RSS  Cp 
393.4122  424.8992 
402.4703  428.8513 
400.4602  427.6921 
400.8952  428.1272 
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APPENDIX  T.  REDUCED  TWO  FACTOR  INTERACTION  MODELS: 

SCATTERPLOTS  WITH  SUPERIMPOSED  REGRESSION 
LINE 


PA 


QA 


Fitted  :  factor(Community)  +  factor(Squadron)  +  1actor(Tot.  Years)  +  lactor(Shift)  +  factor(Community):1actor(S  Fitted  :  (actor(Community)  +  factor(Squadron)  +  factor{Tot. Years)  +  factor(Shift)  +  1actor(Community):<aetor(S 

RS  RM 


iadron)  +  tactor(Rank)  +  factor(Tot.Years)  +  factor(Shop)  +  faetor(Shitt)  +  faclor{Squadron):lac1or(Shitt)  +  faclonadron)  +  factor(Rank)  +  faetor(Tot.  Years)  +  factor(Shop)  +  factor(Shift)  +  iaclor(Squadron):factor(Shift)  +  factor 


cc 


CR 


lity)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot.  Years)  +  factor(Shitt)  +  factor(Community):factor(Shift)  +  faoity)  +  factor(Rank)  +  factor(Tot.Years)  +  factor(Shift)  +  factor{Community):factor(Shift)  +  factor(Rank):factor(Sh 
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APPENDIX  U.  REDUCED  TWO  FACTOR  INTERACTION  MODELS 
HISTOGRAMS  OF  RESIDUALS 
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APPENDIX  V.  REDUCED  TWO  FACTOR  INTERACTION  MODELS: 
RESIDUALS  VERSUS  FITTED  VALUES 

PA  QA 


2.0  2.5  3.0  3.5  4.0  4.S  1  2  3  4 

Fitted  :  factor(Community)  +  faetor(Squadron)  +  factor(Tot.Years)  +  lactor(Shift)  +  (aclor(Conimunity):»actor{S  Fitted  :  factor(Comrminity)  +  factor(Squadron)  +  factor(Tot.Years)  +  factor(Shrft)  +  factor(Community):faetor(S 


RS  RM 


2  3  4  5  1  2  3  4 

ladron)  +  factor(Rank)  +  tactor(Tot.Years)  +  tactor(Shop)  +  factor(Shift)  +  factor(Squadron):factor(Shrft)  +  factonadron)  +  factor(Rank)  +  (actor{Tot.Years)  +  factor(Shop)  +  factor(Shift)  +  factor(Squadron):factor(Shift)  +  factor 


CC  CR 


itty)  +  factor(Squadron)  +  factor(Rank)  +  factor(Tot.Years)  +  factor(Shitl)  +  factor{Community):factor(Shift)  +faoity)  +  faetor(Rank)  +  factor(Tot.Years)  +  factor(Shift)  +  factor(Community):tactor(Shift)  *  factor(Rank):factor(Sh 
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APPENDIX  X.  COMPARISONS  ON  MODELS  USING  ANOVA 


The  values  listed  in  the  following  tables  are  the  probabilities  that  the  models  are 
equivalent.  Values  printed  in  bold  type  indicate  similar  models. 


Component 

Simple  vs 
Two  Factor 
Interaction 

Simple  vs. 
Reduced 
Simple 

Simple  vs. 
Reduced  Two 
Factor 
Interaction 

PA 

0.00316 

0.37994 

0.99999 

QA 

0.00657 

0.17934 

0.99990 

RS 

0.00174 

0.04694 

0.00029 

RM 

0.00730 

0.09862 

1.86442e-7 

CC 

0.00156 

0.20517 

0.00001 

CR 

0.00580 

0.03360 

0.10600 

Component 

Two  Factor 
Interaction 
vs.  Reduced 
Simple 

Two  Factor 
Interaction 
vs.  Reducd 
Two  Factor 
Interaction 

PA 

0.00324 

0.00788 

QA 

0.00488 

0.01596 

RS 

0.00087 

0.01474 

RM 

0.00470 

0.14662 

CC 

0.00121 

0.02202 

CR 

0.00200 

0.02111 

Component 

Reduced 
Simple  vs. 
Reduced  Two 
Factor 
Interaction 

PA 

0.00752 

QA 

0.00120 

RS 

0.00011 

RM 

3:50999e-7 

CC 

0.00008 

CR 

0.00016 
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APPENDIX  Y.  SUBJECT  MATTER  EXPERT  RECOMMMENDED  CHANGES 
TO  MCAS  DEMOGRAPHIC  FACTORS 

Factors  currently  included  in  MCAS  (no  change): 

1.  Community 

2.  Squadron 

3.  Rank 

4.  Work  Center 

5.  Shift 

Items  to  modify  in  MCAS:  * 

Total  years  of  aviation  maintenance  experience  (single  item) 
modified  to: 

6.  Years  worked  in  MOS 

7.  Years  worked  outside  MOS 

Factors  to  add  to  MCAS: 

education/training  level 

8.  attended  an  A  school 

9.  highest  level  of  education  attained 

10.  number  of  maintenance  activities  assigned  to  during  career 
morale/motivation  indicators 

11.  command  advanced 

12.  number  of  personal  awards 

13.  level  of  job  satisfaction 

14.  time  to  attain  qualifications 

15.  past  performance  on  personal  evaluations 

16.  assigned  to  B  tour 

other  factors 

17.  Age 

18.  number  of  months  in  current  squadron 

19.  number  of  deployments 

20.  supervisory  designations  earned 

21.  level  of  confidence  in  Maintenance  Control  Officer 
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